Re: Proposal to retroactively mark materialized views experimental

> > My concerns have consistently been more fundamental - the basic properties > of a theoretically bug-free MV are simultaneously problematic and unknown. > How can we say we have a working feature, when we our definition of > ‘working’ is unknown? Well, we know to some extent that MV's are theor

Re: Proposal to retroactively mark materialized views experimental

Kurt, we seem to be talking past each other. While I am concerned about implementation bugs - which I am certain still exist - I have not at any point raised this issue. My concerns have consistently been more fundamental - the basic properties of a theoretically bug-free MV are simultaneously

Re: Proposal to retroactively mark materialized views experimental

Auth/roles has a non-negligible impact on performance, and isn’t otherwise required by most users. And for some operators getting involved at CQL layer isn’t really viable. If you have hundreds/thousands of clusters and only control the yaml and configs. Adding another flag is something we can

Re: Proposal to retroactively mark materialized views experimental

> > So you’d rather continue to lie to users about the stability of the > feature rather than admitting it was merged in prematurely? It was merged prematurely, but a lot has changed since then and a lot of fixes have been made, and now it's really no more awful than any other component of Cassand

Re: Proposal to retroactively mark materialized views experimental

On Wed, Oct 4, 2017 at 12:23 PM, Jon Haddad wrote: > The default part I was referring to incremental repair. > > SASI still has a pretty fatal issue where nodes OOM: > https://issues.apache.org/jira/browse/CASSANDRA-12662 < > https://issues.apache.org/jira/browse/CASSANDRA-12662> > If you read t

Re: Proposal to retroactively mark materialized views experimental

The default part I was referring to incremental repair. SASI still has a pretty fatal issue where nodes OOM: https://issues.apache.org/jira/browse/CASSANDRA-12662 > On Oct 4, 2017, at 12:21 PM, Pavel Yaskevich wrote: > > On Wed, Oct 4

Re: Proposal to retroactively mark materialized views experimental

On Wed, Oct 4, 2017 at 12:09 PM, Jon Haddad wrote: > MVs work fine for *some use cases*, not the general use case. That’s why > there should be a flag. To opt into the feature when the behavior is only > known to be correct under a certain set of circumstances. Nobody is saying > the flag shou

Re: Proposal to retroactively mark materialized views experimental

MVs work fine for *some use cases*, not the general use case. That’s why there should be a flag. To opt into the feature when the behavior is only known to be correct under a certain set of circumstances. Nobody is saying the flag should be “enable_terrible_feature_nobody_tested_and_we_all_ha

Re: Proposal to retroactively mark materialized views experimental

> > So you’d rather continue to lie to users about the stability of the > feature rather than admitting it was merged in prematurely? Much like w/SASI, this is something that's in the code-base that for > certain use-cases apparently works just fine. I don't know of any outstanding issues with t

Re: Proposal to retroactively mark materialized views experimental

Strongly disagree with MV’s being isolated part. You can feel the touch of the MVs in the read path, write path, metadata handling, whether you use them or not. And comparing any of those before/after MVs were introduced makes me sad every time I face any of it. It made our codebase objectively

Re: Proposal to retroactively mark materialized views experimental

So you’d rather continue to lie to users about the stability of the feature rather than admitting it was merged in prematurely? I’d rather come clean and avoid future problems, and give people the opportunity to stop using MVs rather than let them keep taking risks they’re unaware of. This is

Re: Proposal to retroactively mark materialized views experimental

> > Oh, come on. You're being disingenuous. Not my intent. MV's (and SASI, for example) are fairly well isolated; we have a history of other changes that are much more broadly and higher impact risk-wise across the code-base. If I were an operator and built a critical part of my business on a rel

Re: Proposal to retroactively mark materialized views experimental

Yep. Almost! Changing the default in a minor version might break some scripts/tooling that manipulate schema and potentially create new MVs (as dangerous as it currently is - manipulating schema in that way), and that would still not be very nice. Introducing a flag and leaving it at false in a

Re: Proposal to retroactively mark materialized views experimental

Can't we promote these behavioural flags to keyspace properties (with suitable permissions to edit necessary)? I agree that enabling/disabling features shouldn't require a rolling restart, and nor should switching their consistency safety level. I think this would be the most suitable equivalen

Re: Proposal to retroactively mark materialized views experimental

If "disabling a feature" is just about preventing some CQL from execution along with a warning log message, I'm fine with that. But if that's being the case, I don't really understand why making this change in a minor version would be a problem, since existing MVs wouldn't be affected anyways and s

Re: Proposal to retroactively mark materialized views experimental

Not to detract from the discussion about whether or not to classify X or Y as experimental but https://issues.apache.org/jira/browse/CASSANDRA-8303 was originally about operators preventing users from abusing features (e.g. allow filtering).

Re: Proposal to retroactively mark materialized views experimental

Oh, come on. You're being disingenuous. I invented both algorithms, so I get some say in which is more complex. I fully understand the behaviour of early reopen and can explain it to a lay person in around five minutes. Last time I posted an analysis of MVs it took me several days to get it s

Re: Proposal to retroactively mark materialized views experimental

I agree with Aleksey on all points here. Adding that we should update the docs with warnings about the potential issues with correctness. On Wed, Oct 4, 2017 at 8:25 AM Aleksey Yeshchenko wrote: > We already have those for UDFs and CDC. > > We should have more: for triggers, SASI, and MVs, at lea

Re: Proposal to retroactively mark materialized views experimental

We already have those for UDFs and CDC. We should have more: for triggers, SASI, and MVs, at least. Operators need a way to disable features they haven’t validated. We already have sufficient consensus to introduce the flags, and we should. There also seems to be sufficient consensus on emittin

Re: Proposal to retroactively mark materialized views experimental

I don't agree at face value that early re-open is in sum a lot simpler than MV, or that adding CQL and deprecating Thrift was a lot simpler, or the 8099 refactor, etc. Different types of complexity, certainly, and MV's are arguably harder to prove correct due to surface area of exposure to failure

Re: Proposal to retroactively mark materialized views experimental

So, as the author of one of the disasters you mention (early re-open), I would prefer to learn from the mistake and not repeat it. Unfortunately we seem to be in the habit of repeating it, and that feature was a lot *lot* simpler. Let’s not kid ourselves: MVs are by far and away the most compli

Re: Proposal to retroactively mark materialized views experimental

Introducing feature flags for enabling or disabling different code paths is not sustainable in the long run. It's hard enough to keep up with integration testing with the couple of Jenkins jobs that we have. Running jobs for all permutations of flags that we keep around, would turn out impractical.

Re: Proposal to retroactively mark materialized views experimental

> and providing a feature we don't fully understand, have not fully documented the caveats of, let alone discovered all the problems with nor had that knowledge percolate fully into the wider community. There appear to be varying levels of understanding of the implementation details of MV's (that s

Re: Proposal to retroactively mark materialized views experimental

> > The flag name `cdc_enabled` is simple and, without adjectives, does not > imply "experimental" or "beta" or anything like that. > It does make life easier for both operators and the C* developers. I would be all for a mv_enabled option, assuming it's enabled by default for all existing branche

Re: Proposal to retroactively mark materialized views experimental

> > CDC sounds like it is in the same basket, but it already has the > > `cdc_enabled` yaml flag which defaults false. > > I went this route because I was incredibly wary of changing the CL > code and wanted to shield non-CDC users from any and all risk I > reasonably could. This approach so far

Re: Proposal to retroactively mark materialized views experimental

For the record, in case I was unclear, it was never my intention to suggest that we shouldn't warn about MVs: I would agree that we still should and I'm happy that we do. I would also agree that the remaining caveats and limitations should be more clearly documented. But, I kind of got the feeling

Re: Proposal to retroactively mark materialized views experimental

So, I'm of the opinion there's a difference between users misusing a well understood feature whose shortcomings are widely discussed in the community, and providing a feature we don't fully understand, have not fully documented the caveats of, let alone discovered all the problems with nor had t

Re: Proposal to retroactively mark materialized views experimental

So for some perspective here, how do users who do not get the guarantees of MV’s implement this on their own? They used logged batches. Pseudo CQL here, but you should get the picture: If they don’t ever update data, they do it like so, and it is pretty safe: BEGIN BATCH INSERT tablea blah INSE

Re: Proposal to retroactively mark materialized views experimental

While many users may apparently be using MVs successfully, the problem is how few (if any) know what guarantees they are getting. Since we aren’t even absolutely certain ourselves, it cannot be many. Most of the shortcomings we are aware of are complicated, concern failure scenarios and aren’t

Re: Proposal to retroactively mark materialized views experimental

Lots of users are already using MV's, believe it or not in some cases quite effectively and also on older versions which were still exposed to a lot of the bugs that cause inconsistencies. 3.11.1 has come a long way since then and I think with a bit more documentation around the current issues mark

Re: Proposal to retroactively mark materialized views experimental

I never used the word easier. I think it's a hard problem but it should be our problem if we want people to use our database. I have little opinion of if MVs should be made experimental or opt-in. I'd simply discussing the need for this feature (as opposed to ripping it out)

Re: Proposal to retroactively mark materialized views experimental

> > the default behavioural unsafety we opt for by not writing through the > batch log We always write to the local batchlog (unless the MV replica is the local node)... Most of the bugs have been around tombstones and ttls AFAIK. On Tue, Oct 3, 2017 at 3:23 PM, Benedict Elliott Smith <_...@bel

Re: Proposal to retroactively mark materialized views experimental

This link is a helpful segway to another problem with MVs and defaults - the default behavioural unsafety we opt for by not writing through the batch log, opening far more windows for data inconsistency than the algorithm otherwise permits. Without a way to detect or repair these inconsistencie

Re: Proposal to retroactively mark materialized views experimental

To clarify: > > * There's no way to determine if a view is out of sync with the base table. > * If you do determine that a view is out of sync, the only way to fix it > is to drop and rebuild the view. In this case, 'out of sync' means 'you lost data', since the current design + repair should kee

Re: Proposal to retroactively mark materialized views experimental

Nobody debates that it's easier, the debate is over whether or not it's correct (and more importantly, whether or not people realize it's not strictly correct in all edge cases). Users expect correct results. People are literally betting their jobs on it. When you have to manually manage sync betw

Re: Proposal to retroactively mark materialized views experimental

> > The remaining issues are: > > * There's no way to determine if a view is out of sync with the base table. > * If you do determine that a view is out of sync, the only way to fix it > is to drop and rebuild the view. > * There are liveness issues with updates being reflected in the view. > I ju

Re: Proposal to retroactively mark materialized views experimental

Indeed. Paulo and Zhao did a lot of good work to make the situation less bad. You did some as well. Even I retouched small parts of it - metadata related. I’m sorry if I came off as disrespectful - I didn’t mean to. I’ve seen and I appreciate every commit that went into it. It is however my opi

Re: Proposal to retroactively mark materialized views experimental

Lot's of hard work by folks on MVs and I don't think this proposal is a commentary or reflection on that. What it is, is about signalling to users that this feature has more edge cases and caveats than other tried and true features (like all new features). MVs are still a feature in a "stable" rel

Re: Proposal to retroactively mark materialized views experimental

The remaining issues are: * There's no way to determine if a view is out of sync with the base table. * If you do determine that a view is out of sync, the only way to fix it is to drop and rebuild the view. * There are liveness issues with updates being reflected in the view. On October 3, 2017

Re: Proposal to retroactively mark materialized views experimental

On Tue, Oct 3, 2017 at 5:54 PM, Aleksey Yeshchenko wrote: > There are a couple compromise options here: > > a) Introduce the flag (enalbe_experimental_features, or maybe one per > experimental feature), set it to ‘false’ in the yaml, but have the default be > ‘true’. So that if you are upgrading

Re: Proposal to retroactively mark materialized views experimental

Thanks for bringing this up Kurt, it is a fair point. Given the work that Paulo and Zhao have done to get MV’s in shape, what are the outstanding issues that would warrant making them experimental? > On Oct 3, 2017, at 5:56 AM, kurt greaves wrote: > > And finally, back onto the original top

Re: Proposal to retroactively mark materialized views experimental

There are a couple compromise options here: a) Introduce the flag (enalbe_experimental_features, or maybe one per experimental feature), set it to ‘false’ in the yaml, but have the default be ‘true’. So that if you are upgrading from a previous minor to the next without updating the yaml, you n

Re: Proposal to retroactively mark materialized views experimental

> there was "some" reason that even major changes had to be > squeezed into 3.0 before it was released The TL;DR is: having One Version to Rule Them All forces a slew of changes into majors only, since bumping the MessagingService Version has far-reaching impacts. Reference: https://issues.apache.o

Re: Proposal to retroactively mark materialized views experimental

Well this is all terribly interesting. I was actually going to get some discussion going about this during my talk, which unfortunately didn't happen, but I'll take this opportunity to push my agenda. My 99 cents: *tl;dr: we should probably just focus on not releasing completely broken features in

Re: Proposal to retroactively mark materialized views experimental

> CDC sounds like it is in the same basket, but it already has the > `cdc_enabled` yaml flag which defaults false. I went this route because I was incredibly wary of changing the CL code and wanted to shield non-CDC users from any and all risk I reasonably could. I don't know of any outstanding is

Re: Proposal to retroactively mark materialized views experimental

Just one more data point, but I personally don't feel that disabling new MV creation (or new SAS index creation for that matter) by default _in a patch release_ is terribly nice. There can absolutely be code out there that creates MV/SASI indexes somewhat automatically on some events and it would b

Re: Proposal to retroactively mark materialized views experimental

At the risk of sounding redundant it sounds like for MVs at this point we want to preserve current functionality for existing users. We would have a flag in the yaml to disable it by default for new MV creation with an error message. In addition we want a warning in the log and in cqlsh upon cr

Re: Proposal to retroactively mark materialized views experimental

On 3 October 2017 at 04:57, Aleksey Yeshchenko wrote: > The idea is to check the flag in CreateViewStatement, so creation of new > MVs doesn’t succeed without that flag flipped. > Obviously, just disabling existing MVs working in a minor would be silly. > As for the warning - yes, that should als

Re: Proposal to retroactively mark materialized views experimental

Experimental / feature flags in the yaml file is a far better choice for operators. Explicit opt-in for "dangerous" features would greatly help protect new users as well. Plenty of users ignore batch size warnings, tombstone warnings, etc. A soft warnings only approach would not achieve the goals

Re: Proposal to retroactively mark materialized views experimental

If a user (vs Cassandra dev) perspective is welcome - I'd recommend similarly identifying experimental features in the DESCRIBE / DESC cqlsh output as well. On Mon, Oct 2, 2017 at 4:21 PM, Jeremiah D Jordan wrote: > Blake, > We are not saying to just put something in logs, we are talking about t

Re: Proposal to retroactively mark materialized views experimental

Yep. And that would be nice to have in addition to the opt-in flag in the yaml for the operators that’s stricter than a warning. — AY On 2 October 2017 at 22:21:33, Jeremiah D Jordan (jerem...@datastax.com) wrote: We are not saying to just put something in logs, we are talking about the warn a

Re: Proposal to retroactively mark materialized views experimental

Yes, I understand what you're saying. The points I'm making about logs still apply. It's possible for drivers and object mappers to handle queries and schema changes, and have developers rarely open cqlsh. It's also not uncommon for schema changes to be done by a different group than the develop

Re: Proposal to retroactively mark materialized views experimental

Blake, We are not saying to just put something in logs, we are talking about the warn actually showing up in cqlsh. When you issue a native protocol warn cqlsh will print it out on the console in front of you in the results of the query. https://issues.apache.org/jira/browse/CASSANDRA-8930

Re: Proposal to retroactively mark materialized views experimental

It is different because it allows the operators to set those boundaries rather than rely on users doing the right thing. So ideally you’d have the flag (and given the state of MVs, both design and the implementation, I’d argue the default should be NOPE) - for the operators, and a warning in cq

Re: Proposal to retroactively mark materialized views experimental

Developers are also not the only people that are able to make decisions. Keeping it in the YAML means an operator can disable it vs a developer *maybe* seeing the warning. Keep in mind not everyone creates tables through CQLSH. > On Oct 2, 2017, at 2:05 PM, Blake Eggleston wrote: > > The mes

Re: Proposal to retroactively mark materialized views experimental

The message isn't materially different, but it will reach fewer people, later. People typically aren't as attentive to logs as they should be. Developers finding out about new warnings in the logs later than they could have, sometimes even after it's been deployed, is not uncommon. It's happened

Re: Proposal to retroactively mark materialized views experimental

"Nobody is talking about removing MVs." Not precisely true for this email thread: "but should there be some point in the future where we consider removing them from the code base unless they have gotten significant improvement as well?" IMO a .yaml change requirement isn't materially different th

Re: Proposal to retroactively mark materialized views experimental

Ok so IF there is a flag to enable MV (à-la UDA/UDF in cassandra.yaml) then I'm fine with it. I initially understood that we wanted to disable it definitively. Maybe we should then add an explicit error message when MV is disabled and someone tries to use it, something like: "MV has been disabled,

Re: Proposal to retroactively mark materialized views experimental

There’s a big difference between removal of a protocol that every single C* user had to use and disabling a feature which is objectively broken and almost nobody is using. Nobody is talking about removing MVs. If you want to use them you can enable them very trivially, but it should be an expl

Re: Proposal to retroactively mark materialized views experimental

"I would (in a patch release) disable MV CREATE statements, and emit warnings for ALTER statements and on schema load if they’re not explicitly enabled" --> I find this pretty extreme. Now we have an existing feature sitting there in the base code but forbidden from version xxx onward. Since when

Re: Proposal to retroactively mark materialized views experimental

> Only emitting a warning really reduces visibility where we need it: in the > development process. How does emitting a native protocol warning reduce visibility during the development process? If you run CREATE MV and cqlsh then prints out a giant warning statement about how it is an experime

Re: Proposal to retroactively mark materialized views experimental

Yeah, I'm not proposing that we disable MVs in existing clusters. On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (alek...@apple.com) wrote: The idea is to check the flag in CreateViewStatement, so creation of new MVs doesn’t succeed without that flag flipped. Obviously, just disabling

Re: Proposal to retroactively mark materialized views experimental

Having now helped a few folks that have put MVs into prod without realizing what they got themselves into, I’m +1 on a flag disabling the feature by default. A WARN message would not have helped them. > On Oct 2, 2017, at 10:56 AM, Blake Eggleston wrote: > > Yeah I’m not sure that just emitt

Re: Proposal to retroactively mark materialized views experimental

The idea is to check the flag in CreateViewStatement, so creation of new MVs doesn’t succeed without that flag flipped. Obviously, just disabling existing MVs working in a minor would be silly. As for the warning - yes, that should also be emitted. Unconditionally. — AY On 2 October 2017 at 18

Re: Proposal to retroactively mark materialized views experimental

Yeah I’m not sure that just emitting a warning is enough. The point is to be super explicit that bad things will happen if you use MVs. I would (in a patch release) disable MV CREATE statements, and emit warnings for ALTER statements and on schema load if they’re not explicitly enabled. Only emi

Re: Proposal to retroactively mark materialized views experimental

Hindsight is 20/20. For 8099 this is the reason we cut the 2.2 release before 8099 got merged. But moving forward with where we are now, if we are going to start adding some experimental flags to things, then I would definitely put SASI on this list as well. For both SASI and MV I don’t know

Re: Proposal to retroactively mark materialized views experimental

> > I think committing 8099, or at the very least, parts of it, behind an > experimental flag would have been the right thing to do. With a major refactor like that, it's a staggering amount of extra work to have a parallel re-write of core components of a storage engine accessible in parallel to

Re: Proposal to retroactively mark materialized views experimental

triggers On 10/01/2017 11:25 AM, Jeff Jirsa wrote: Historical examples are anything that you wouldn’t bet your job on for the first release: Udf/uda in 2.2 Incremental repair - would have yanked the flag following 9143 SASI - probably still experimental Counters - all sorts of correctness iss

Re: Proposal to retroactively mark materialized views experimental

I think you're presenting a false dichotomy here. Yes there are people who are not interested in taking risks with C* and are still running 1.2, there are probably a few people who would put trunk in prod if we packaged it up for them, but there's a whole spectrum of users in between. Operator c

Re: Proposal to retroactively mark materialized views experimental

So basically we're saying that even with a lot of tests, you're never sure to cover all the possible edge cases and the real stamp for "production readiness" is only when the "experimental features" have been deployed in various clusters with various scenarios/use-cases, just re-phrasing Blake here

Re: Proposal to retroactively mark materialized views experimental

I'm not sure the main issue in the case of MVs is testing. In this case it seems to be that there are some design issues and/or the design was only works in some overly restrictive use cases. That MVs were committed knowing these were issues seems to be the real problem. So in the case of MVs, s

Re: Proposal to retroactively mark materialized views experimental

Historical examples are anything that you wouldn’t bet your job on for the first release: Udf/uda in 2.2 Incremental repair - would have yanked the flag following 9143 SASI - probably still experimental Counters - all sorts of correctness issues originally, no longer true since the rewrite in 2

Re: Proposal to retroactively mark materialized views experimental

How should we transition one feature from the "experimental" state to "production ready" state ? On which criteria ? On Sun, Oct 1, 2017 at 12:12 PM, Marcus Eriksson wrote: > I was just thinking that we should try really hard to avoid adding > experimental features - they are experimental due

Re: Proposal to retroactively mark materialized views experimental

I was just thinking that we should try really hard to avoid adding experimental features - they are experimental due to lack of testing right? There should be a clear path to making the feature non-experimental (or get it removed) and having that path discussed on dev@ might give more visibility to

Re: Proposal to retroactively mark materialized views experimental

+1 to marking it experimental and also making it a prop to enable these features. By default I think they should be disabled. On Fri, Sep 29, 2017 at 10:42 PM, Jeff Jirsa wrote: > Reviewers should be able to suggest when experimental is warranted, and > conversation on dev+jira to justify when i

Re: Proposal to retroactively mark materialized views experimental

Reviewers should be able to suggest when experimental is warranted, and conversation on dev+jira to justify when it’s transitioned from experimental to stable? We should remove the flag as soon as we’re (collectively) confident in a feature’s behavior - at least correctness, if not performance.

Re: Proposal to retroactively mark materialized views experimental

+1 on marking MVs experimental, but should there be some point in the future where we consider removing them from the code base unless they have gotten significant improvement as well? We probably need to enforce some kind of process for adding new experimental features in the future - perhaps a m

Re: Proposal to retroactively mark materialized views experimental

We tried perf testing MVs internally here but did not see good results with it, hence paused its usage. +1 on tagging certain features which are not PROD ready or not stable enough. Regards, Vinay Chella On Fri, Sep 29, 2017 at 7:22 PM, Ben Bromhead wrote: > I'm a fan of introducing experimenta

Re: Proposal to retroactively mark materialized views experimental

+1, and as Ben said, we should be willing to consider other things that deserve experimental flags as well (SASI comes to mind as a potential example) On Fri, Sep 29, 2017 at 11:09 AM, Blake Eggleston wrote: > Hi dev@, > > I’d like to propose that we retroactively classify materialized views as

Re: Proposal to retroactively mark materialized views experimental

I'm a fan of introducing experimental flags in general as well, +1 On Fri, 29 Sep 2017 at 13:22 Jon Haddad wrote: > I’m very much +1 on this, and to new features in general. > > I think having a clear line in which we classify something as production > ready would be nice. It would be great i

Re: Proposal to retroactively mark materialized views experimental

I’m very much +1 on this, and to new features in general. I think having a clear line in which we classify something as production ready would be nice. It would be great if committers were using the feature in prod and could vouch for it’s stability. > On Sep 29, 2017, at 1:09 PM, Blake Eggl

Proposal to retroactively mark materialized views experimental

Hi dev@, I’d like to propose that we retroactively classify materialized views as an experimental feature, disable them by default, and require users to enable them through a config setting before using. Materialized views have several issues that make them (effectively) unusable in production