Re: Proposal to retroactively mark materialized views experimental

> > So you’d rather continue to lie to users about the stability of the > feature rather than admitting it was merged in prematurely? It was merged prematurely, but a lot has changed since then and a lot of fixes have been made, and now it's really no more awful than any other component of

Re: Proposal to retroactively mark materialized views experimental

On Wed, Oct 4, 2017 at 12:23 PM, Jon Haddad wrote: > The default part I was referring to incremental repair. > > SASI still has a pretty fatal issue where nodes OOM: > https://issues.apache.org/jira/browse/CASSANDRA-12662 < >

Re: Proposal to retroactively mark materialized views experimental

The default part I was referring to incremental repair. SASI still has a pretty fatal issue where nodes OOM: https://issues.apache.org/jira/browse/CASSANDRA-12662 > On Oct 4, 2017, at 12:21 PM, Pavel Yaskevich

Re: Proposal to retroactively mark materialized views experimental

On Wed, Oct 4, 2017 at 12:09 PM, Jon Haddad wrote: > MVs work fine for *some use cases*, not the general use case. That’s why > there should be a flag. To opt into the feature when the behavior is only > known to be correct under a certain set of circumstances. Nobody is

Re: Proposal to retroactively mark materialized views experimental

MVs work fine for *some use cases*, not the general use case. That’s why there should be a flag. To opt into the feature when the behavior is only known to be correct under a certain set of circumstances. Nobody is saying the flag should be

Re: Proposal to retroactively mark materialized views experimental

> > So you’d rather continue to lie to users about the stability of the > feature rather than admitting it was merged in prematurely? Much like w/SASI, this is something that's in the code-base that for > certain use-cases apparently works just fine. I don't know of any outstanding issues with

Re: Proposal to retroactively mark materialized views experimental

Strongly disagree with MV’s being isolated part. You can feel the touch of the MVs in the read path, write path, metadata handling, whether you use them or not. And comparing any of those before/after MVs were introduced makes me sad every time I face any of it. It made our codebase

Re: Proposal to retroactively mark materialized views experimental

So you’d rather continue to lie to users about the stability of the feature rather than admitting it was merged in prematurely? I’d rather come clean and avoid future problems, and give people the opportunity to stop using MVs rather than let them keep taking risks they’re unaware of. This is

Re: Proposal to retroactively mark materialized views experimental

> > Oh, come on. You're being disingenuous. Not my intent. MV's (and SASI, for example) are fairly well isolated; we have a history of other changes that are much more broadly and higher impact risk-wise across the code-base. If I were an operator and built a critical part of my business on a

Re: Proposal to retroactively mark materialized views experimental

Yep. Almost! Changing the default in a minor version might break some scripts/tooling that manipulate schema and potentially create new MVs (as dangerous as it currently is - manipulating schema in that way), and that would still not be very nice. Introducing a flag and leaving it at false in

Re: Proposal to retroactively mark materialized views experimental

I don't agree at face value that early re-open is in sum a lot simpler than MV, or that adding CQL and deprecating Thrift was a lot simpler, or the 8099 refactor, etc. Different types of complexity, certainly, and MV's are arguably harder to prove correct due to surface area of exposure to failure

Re: Proposal to retroactively mark materialized views experimental

Introducing feature flags for enabling or disabling different code paths is not sustainable in the long run. It's hard enough to keep up with integration testing with the couple of Jenkins jobs that we have. Running jobs for all permutations of flags that we keep around, would turn out

Re: Proposal to retroactively mark materialized views experimental

> and providing a feature we don't fully understand, have not fully documented the caveats of, let alone discovered all the problems with nor had that knowledge percolate fully into the wider community. There appear to be varying levels of understanding of the implementation details of MV's (that

Re: Proposal to retroactively mark materialized views experimental

> > The flag name `cdc_enabled` is simple and, without adjectives, does not > imply "experimental" or "beta" or anything like that. > It does make life easier for both operators and the C* developers. I would be all for a mv_enabled option, assuming it's enabled by default for all existing

Re: Proposal to retroactively mark materialized views experimental

> > CDC sounds like it is in the same basket, but it already has the > > `cdc_enabled` yaml flag which defaults false. > > I went this route because I was incredibly wary of changing the CL > code and wanted to shield non-CDC users from any and all risk I > reasonably could. This approach so far

Re: Proposal to retroactively mark materialized views experimental

For the record, in case I was unclear, it was never my intention to suggest that we shouldn't warn about MVs: I would agree that we still should and I'm happy that we do. I would also agree that the remaining caveats and limitations should be more clearly documented. But, I kind of got the

Re: Proposal to retroactively mark materialized views experimental

So, I'm of the opinion there's a difference between users misusing a well understood feature whose shortcomings are widely discussed in the community, and providing a feature we don't fully understand, have not fully documented the caveats of, let alone discovered all the problems with nor had

Re: Proposal to retroactively mark materialized views experimental

So for some perspective here, how do users who do not get the guarantees of MV’s implement this on their own? They used logged batches. Pseudo CQL here, but you should get the picture: If they don’t ever update data, they do it like so, and it is pretty safe: BEGIN BATCH INSERT tablea blah

Re: Proposal to retroactively mark materialized views experimental

While many users may apparently be using MVs successfully, the problem is how few (if any) know what guarantees they are getting. Since we aren’t even absolutely certain ourselves, it cannot be many. Most of the shortcomings we are aware of are complicated, concern failure scenarios and

Re: Proposal to retroactively mark materialized views experimental

Lots of users are already using MV's, believe it or not in some cases quite effectively and also on older versions which were still exposed to a lot of the bugs that cause inconsistencies. 3.11.1 has come a long way since then and I think with a bit more documentation around the current issues

Re: Proposal to retroactively mark materialized views experimental

I never used the word easier. I think it's a hard problem but it should be our problem if we want people to use our database. I have little opinion of if MVs should be made experimental or opt-in. I'd simply discussing the need for this feature (as opposed to ripping it out)

Re: Proposal to retroactively mark materialized views experimental

> > the default behavioural unsafety we opt for by not writing through the > batch log We always write to the local batchlog (unless the MV replica is the local node)... Most of the bugs have been around tombstones and ttls AFAIK. On Tue, Oct 3, 2017 at 3:23 PM, Benedict Elliott Smith

Re: Proposal to retroactively mark materialized views experimental

This link is a helpful segway to another problem with MVs and defaults - the default behavioural unsafety we opt for by not writing through the batch log, opening far more windows for data inconsistency than the algorithm otherwise permits. Without a way to detect or repair these

Re: Proposal to retroactively mark materialized views experimental

To clarify: > > * There's no way to determine if a view is out of sync with the base table. > * If you do determine that a view is out of sync, the only way to fix it > is to drop and rebuild the view. In this case, 'out of sync' means 'you lost data', since the current design + repair should

Re: Proposal to retroactively mark materialized views experimental

Nobody debates that it's easier, the debate is over whether or not it's correct (and more importantly, whether or not people realize it's not strictly correct in all edge cases). Users expect correct results. People are literally betting their jobs on it. When you have to manually manage sync

Re: Proposal to retroactively mark materialized views experimental

> > The remaining issues are: > > * There's no way to determine if a view is out of sync with the base table. > * If you do determine that a view is out of sync, the only way to fix it > is to drop and rebuild the view. > * There are liveness issues with updates being reflected in the view. > I

Re: Proposal to retroactively mark materialized views experimental

Indeed. Paulo and Zhao did a lot of good work to make the situation less bad. You did some as well. Even I retouched small parts of it - metadata related. I’m sorry if I came off as disrespectful - I didn’t mean to. I’ve seen and I appreciate every commit that went into it. It is however my

Re: Proposal to retroactively mark materialized views experimental

The remaining issues are: * There's no way to determine if a view is out of sync with the base table. * If you do determine that a view is out of sync, the only way to fix it is to drop and rebuild the view. * There are liveness issues with updates being reflected in the view. On October 3,

Re: Proposal to retroactively mark materialized views experimental

On Tue, Oct 3, 2017 at 5:54 PM, Aleksey Yeshchenko wrote: > There are a couple compromise options here: > > a) Introduce the flag (enalbe_experimental_features, or maybe one per > experimental feature), set it to ‘false’ in the yaml, but have the default be > ‘true’. So that

Re: Proposal to retroactively mark materialized views experimental

There are a couple compromise options here: a) Introduce the flag (enalbe_experimental_features, or maybe one per experimental feature), set it to ‘false’ in the yaml, but have the default be ‘true’. So that if you are upgrading from a previous minor to the next without updating the yaml, you

Re: Proposal to retroactively mark materialized views experimental

> there was "some" reason that even major changes had to be > squeezed into 3.0 before it was released The TL;DR is: having One Version to Rule Them All forces a slew of changes into majors only, since bumping the MessagingService Version has far-reaching impacts. Reference:

Re: Proposal to retroactively mark materialized views experimental

Well this is all terribly interesting. I was actually going to get some discussion going about this during my talk, which unfortunately didn't happen, but I'll take this opportunity to push my agenda. My 99 cents: *tl;dr: we should probably just focus on not releasing completely broken features

Re: Proposal to retroactively mark materialized views experimental

> CDC sounds like it is in the same basket, but it already has the > `cdc_enabled` yaml flag which defaults false. I went this route because I was incredibly wary of changing the CL code and wanted to shield non-CDC users from any and all risk I reasonably could. I don't know of any outstanding

Re: Proposal to retroactively mark materialized views experimental

At the risk of sounding redundant it sounds like for MVs at this point we want to preserve current functionality for existing users. We would have a flag in the yaml to disable it by default for new MV creation with an error message. In addition we want a warning in the log and in cqlsh upon

Re: Proposal to retroactively mark materialized views experimental

On 3 October 2017 at 04:57, Aleksey Yeshchenko wrote: > The idea is to check the flag in CreateViewStatement, so creation of new > MVs doesn’t succeed without that flag flipped. > Obviously, just disabling existing MVs working in a minor would be silly. > As for the warning -

Re: Proposal to retroactively mark materialized views experimental

Experimental / feature flags in the yaml file is a far better choice for operators. Explicit opt-in for "dangerous" features would greatly help protect new users as well. Plenty of users ignore batch size warnings, tombstone warnings, etc. A soft warnings only approach would not achieve the goals

Re: Proposal to retroactively mark materialized views experimental

If a user (vs Cassandra dev) perspective is welcome - I'd recommend similarly identifying experimental features in the DESCRIBE / DESC cqlsh output as well. On Mon, Oct 2, 2017 at 4:21 PM, Jeremiah D Jordan wrote: > Blake, > We are not saying to just put something in

Re: Proposal to retroactively mark materialized views experimental

Yep. And that would be nice to have in addition to the opt-in flag in the yaml for the operators that’s stricter than a warning. — AY On 2 October 2017 at 22:21:33, Jeremiah D Jordan (jerem...@datastax.com) wrote: We are not saying to just put something in logs, we are talking about the warn

Re: Proposal to retroactively mark materialized views experimental

Yes, I understand what you're saying. The points I'm making about logs still apply. It's possible for drivers and object mappers to handle queries and schema changes, and have developers rarely open cqlsh. It's also not uncommon for schema changes to be done by a different group than the

Re: Proposal to retroactively mark materialized views experimental

Blake, We are not saying to just put something in logs, we are talking about the warn actually showing up in cqlsh. When you issue a native protocol warn cqlsh will print it out on the console in front of you in the results of the query. https://issues.apache.org/jira/browse/CASSANDRA-8930

Re: Proposal to retroactively mark materialized views experimental

It is different because it allows the operators to set those boundaries rather than rely on users doing the right thing. So ideally you’d have the flag (and given the state of MVs, both design and the implementation, I’d argue the default should be NOPE) - for the operators, and a warning in

Re: Proposal to retroactively mark materialized views experimental

Developers are also not the only people that are able to make decisions. Keeping it in the YAML means an operator can disable it vs a developer *maybe* seeing the warning. Keep in mind not everyone creates tables through CQLSH. > On Oct 2, 2017, at 2:05 PM, Blake Eggleston

Re: Proposal to retroactively mark materialized views experimental

The message isn't materially different, but it will reach fewer people, later. People typically aren't as attentive to logs as they should be. Developers finding out about new warnings in the logs later than they could have, sometimes even after it's been deployed, is not uncommon. It's

Re: Proposal to retroactively mark materialized views experimental

"Nobody is talking about removing MVs." Not precisely true for this email thread: "but should there be some point in the future where we consider removing them from the code base unless they have gotten significant improvement as well?" IMO a .yaml change requirement isn't materially different

Re: Proposal to retroactively mark materialized views experimental

Ok so IF there is a flag to enable MV (à-la UDA/UDF in cassandra.yaml) then I'm fine with it. I initially understood that we wanted to disable it definitively. Maybe we should then add an explicit error message when MV is disabled and someone tries to use it, something like: "MV has been

Re: Proposal to retroactively mark materialized views experimental

There’s a big difference between removal of a protocol that every single C* user had to use and disabling a feature which is objectively broken and almost nobody is using. Nobody is talking about removing MVs. If you want to use them you can enable them very trivially, but it should be an

Re: Proposal to retroactively mark materialized views experimental

"I would (in a patch release) disable MV CREATE statements, and emit warnings for ALTER statements and on schema load if they’re not explicitly enabled" --> I find this pretty extreme. Now we have an existing feature sitting there in the base code but forbidden from version xxx onward. Since

Re: Proposal to retroactively mark materialized views experimental

> Only emitting a warning really reduces visibility where we need it: in the > development process. How does emitting a native protocol warning reduce visibility during the development process? If you run CREATE MV and cqlsh then prints out a giant warning statement about how it is an

Re: Proposal to retroactively mark materialized views experimental

Yeah, I'm not proposing that we disable MVs in existing clusters. On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (alek...@apple.com) wrote: The idea is to check the flag in CreateViewStatement, so creation of new MVs doesn’t succeed without that flag flipped. Obviously, just

Re: Proposal to retroactively mark materialized views experimental

Having now helped a few folks that have put MVs into prod without realizing what they got themselves into, I’m +1 on a flag disabling the feature by default. A WARN message would not have helped them. > On Oct 2, 2017, at 10:56 AM, Blake Eggleston wrote: > > Yeah I’m

Re: Proposal to retroactively mark materialized views experimental

The idea is to check the flag in CreateViewStatement, so creation of new MVs doesn’t succeed without that flag flipped. Obviously, just disabling existing MVs working in a minor would be silly. As for the warning - yes, that should also be emitted. Unconditionally. — AY On 2 October 2017 at

Re: Proposal to retroactively mark materialized views experimental

Yeah I’m not sure that just emitting a warning is enough. The point is to be super explicit that bad things will happen if you use MVs. I would (in a patch release) disable MV CREATE statements, and emit warnings for ALTER statements and on schema load if they’re not explicitly enabled. Only

Re: Proposal to retroactively mark materialized views experimental

Hindsight is 20/20. For 8099 this is the reason we cut the 2.2 release before 8099 got merged. But moving forward with where we are now, if we are going to start adding some experimental flags to things, then I would definitely put SASI on this list as well. For both SASI and MV I don’t know

Re: Proposal to retroactively mark materialized views experimental

triggers On 10/01/2017 11:25 AM, Jeff Jirsa wrote: Historical examples are anything that you wouldn’t bet your job on for the first release: Udf/uda in 2.2 Incremental repair - would have yanked the flag following 9143 SASI - probably still experimental Counters - all sorts of correctness

Re: Proposal to retroactively mark materialized views experimental

I think you're presenting a false dichotomy here. Yes there are people who are not interested in taking risks with C* and are still running 1.2, there are probably a few people who would put trunk in prod if we packaged it up for them, but there's a whole spectrum of users in between. Operator

Re: Proposal to retroactively mark materialized views experimental

So basically we're saying that even with a lot of tests, you're never sure to cover all the possible edge cases and the real stamp for "production readiness" is only when the "experimental features" have been deployed in various clusters with various scenarios/use-cases, just re-phrasing Blake

Re: Proposal to retroactively mark materialized views experimental

I'm not sure the main issue in the case of MVs is testing. In this case it seems to be that there are some design issues and/or the design was only works in some overly restrictive use cases. That MVs were committed knowing these were issues seems to be the real problem. So in the case of MVs,

Re: Proposal to retroactively mark materialized views experimental

Historical examples are anything that you wouldn’t bet your job on for the first release: Udf/uda in 2.2 Incremental repair - would have yanked the flag following 9143 SASI - probably still experimental Counters - all sorts of correctness issues originally, no longer true since the rewrite in

Re: Proposal to retroactively mark materialized views experimental

How should we transition one feature from the "experimental" state to "production ready" state ? On which criteria ? On Sun, Oct 1, 2017 at 12:12 PM, Marcus Eriksson wrote: > I was just thinking that we should try really hard to avoid adding > experimental features - they

Re: Proposal to retroactively mark materialized views experimental

I was just thinking that we should try really hard to avoid adding experimental features - they are experimental due to lack of testing right? There should be a clear path to making the feature non-experimental (or get it removed) and having that path discussed on dev@ might give more visibility

Re: Proposal to retroactively mark materialized views experimental

+1 to marking it experimental and also making it a prop to enable these features. By default I think they should be disabled. On Fri, Sep 29, 2017 at 10:42 PM, Jeff Jirsa wrote: > Reviewers should be able to suggest when experimental is warranted, and > conversation on

Re: Proposal to retroactively mark materialized views experimental

Reviewers should be able to suggest when experimental is warranted, and conversation on dev+jira to justify when it’s transitioned from experimental to stable? We should remove the flag as soon as we’re (collectively) confident in a feature’s behavior - at least correctness, if not

Re: Proposal to retroactively mark materialized views experimental

+1 on marking MVs experimental, but should there be some point in the future where we consider removing them from the code base unless they have gotten significant improvement as well? We probably need to enforce some kind of process for adding new experimental features in the future - perhaps a

Re: Proposal to retroactively mark materialized views experimental

We tried perf testing MVs internally here but did not see good results with it, hence paused its usage. +1 on tagging certain features which are not PROD ready or not stable enough. Regards, Vinay Chella On Fri, Sep 29, 2017 at 7:22 PM, Ben Bromhead wrote: > I'm a fan of

Re: Proposal to retroactively mark materialized views experimental

+1, and as Ben said, we should be willing to consider other things that deserve experimental flags as well (SASI comes to mind as a potential example) On Fri, Sep 29, 2017 at 11:09 AM, Blake Eggleston wrote: > Hi dev@, > > I’d like to propose that we retroactively classify

Re: Proposal to retroactively mark materialized views experimental

I'm a fan of introducing experimental flags in general as well, +1 On Fri, 29 Sep 2017 at 13:22 Jon Haddad wrote: > I’m very much +1 on this, and to new features in general. > > I think having a clear line in which we classify something as production > ready would be nice.

Re: Proposal to retroactively mark materialized views experimental

I’m very much +1 on this, and to new features in general. I think having a clear line in which we classify something as production ready would be nice. It would be great if committers were using the feature in prod and could vouch for it’s stability. > On Sep 29, 2017, at 1:09 PM, Blake