Re: [VOTE] Release dtest-api 0.0.16

[Blake Eggleston]

+1On Aug 17, 2023, at 12:37 AM, Alex Petrov  wrote:+1On Thu, Aug 17, 2023, at 4:46 AM, Brandon Williams wrote:+1Kind Regards,BrandonOn Wed, Aug 16, 2023 at 4:34 PM Dinesh Joshi  wrote:>> Proposing the test build of in-jvm dtest API 0.0.16 for release.>> Repository:> https://gitbox.apache.org/repos/asf?p=cassandra-in-jvm-dtest-api.git>> Candidate SHA:> https://github.com/apache/cassandra-in-jvm-dtest-api/commit/1ba6ef93d0721741b5f6d6d72cba3da03fe78438> tagged with 0.0.16>> Artifacts:> https://repository.apache.org/content/repositories/orgapachecassandra-1307/org/apache/cassandra/dtest-api/0.0.16/>> Key signature: 53371F9B1B425A336988B6A03B6042413D323470>> Changes since last release:>> * CASSANDRA-18727 - JMXUtil.getJmxConnector should retry connection attempts>> The vote will be open for 24 hours. Everyone who has tested the build> is invited to vote. Votes by PMC members are considered binding. A> vote passes if there are at least three binding +1s.>

Re: [VOTE] CEP-34: mTLS based client and internode authenticators

[Blake Eggleston]

+1

> On Jul 21, 2023, at 9:57 AM, Jyothsna Konisa  wrote:
> 
> Hi Everyone!
> 
> I would like to start a vote thread for CEP-34.
> 
> Proposal: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-34%3A+mTLS+based+client+and+internode+authenticators
> JIRA   : 
> https://issues.apache.org/jira/browse/CASSANDRA-18554
> Draft Implementation : https://github.com/apache/cassandra/pull/2372
> Discussion : 
> https://lists.apache.org/thread/pnfg65r76rbbs70hwhsz94ds6yo2042f
> 
> The vote will be open for 72 hours. A vote passes if there are at least 3 
> binding +1s and no binding vetoes.
> 
> Thanks,
> Jyothsna Konisa.

Re: [VOTE] CEP-21 Transactional Cluster Metadata

[Blake Eggleston]

+1

> On Feb 6, 2023, at 8:15 AM, Sam Tunnicliffe  wrote:
> 
> Hi everyone,
> 
> I would like to start a vote on this CEP.
> 
> Proposal:
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata
> 
> Discussion:
> https://lists.apache.org/thread/h25skwkbdztz9hj2pxtgh39rnjfzckk7
> 
> The vote will be open for 72 hours.
> A vote passes if there are at least three binding +1s and no binding vetoes.
> 
> Thanks,
> Sam

Re: [DISCUSS] Removing support for java 8

[Blake Eggleston]

Let me look into how much work it would take to get accord building with jdk8, 
that may create the least amount of work for everyone involved.

> On Aug 29, 2022, at 2:42 PM, Ekaterina Dimitrova  
> wrote:
> 
> We will need also Jenkins, cassandra-builds. We need to add also j11 upgrade 
> tests and fix some other inconsistencies what jobs we run. I have it on my 
> list and some things partially ready. And I am sure I am missing something. 
> As I said in Slack, I am not against, just trying to be optimal and creating 
> less noise when/where it is possible and be efficient so that people on all 
> ends can progress with whatever they work on. Let’s talk to Mick and put down 
> the pin-points and assess the plan? How about that? 
> 
> On Mon, 29 Aug 2022 at 17:25, Blake Eggleston  <mailto:beggles...@apple.com>> wrote:
> Can you say some more about what extra work you expect removing jdk8 to 
> cause? I'd expect removing jdk8 to be mostly subtractive (from build.xml and 
> circleci confs), and jdk17 support to be mostly additive. 
> 
> We're getting ready to merge the first set of accord integration patches, and 
> the accord library requires java11+. My understanding is that jdk17 support 
> in C* is still several months away, so if we can avoid making one a 
> dependency on the other, that would be preferable.
> 
> 
>> On Aug 29, 2022, at 2:01 PM, Brandon Williams > <mailto:dri...@gmail.com>> wrote:
>> 
>> +1 for removing it when we add 17, to avoid making extra work.
>> 
>> Kind Regards,
>> Brandon
>> 
>> On Mon, Aug 29, 2022 at 3:40 PM Blake Eggleston > <mailto:beggles...@apple.com>> wrote:
>>> 
>>> Sorry, I meant trunk, not 4.1 :)
>>> 
>>>> On Aug 29, 2022, at 1:09 PM, Blake Eggleston >>> <mailto:beggles...@apple.com>> wrote:
>>>> 
>>>> Hi all, I wanted to propose removing jdk8 support for 4.1. Active support 
>>>> ended back in March of this year, and I believe the community has built 
>>>> enough confidence in java 11 to make it an uncontroversial change for our 
>>>> next major release. Let me know what you think.
>>>> 
>>>> Thanks,
>>>> 
>>>> Blake
>>> 
> 

Re: [DISCUSS] Removing support for java 8

[Blake Eggleston]

Can you say some more about what extra work you expect removing jdk8 to cause? 
I'd expect removing jdk8 to be mostly subtractive (from build.xml and circleci 
confs), and jdk17 support to be mostly additive. 

We're getting ready to merge the first set of accord integration patches, and 
the accord library requires java11+. My understanding is that jdk17 support in 
C* is still several months away, so if we can avoid making one a dependency on 
the other, that would be preferable.


> On Aug 29, 2022, at 2:01 PM, Brandon Williams  wrote:
> 
> +1 for removing it when we add 17, to avoid making extra work.
> 
> Kind Regards,
> Brandon
> 
> On Mon, Aug 29, 2022 at 3:40 PM Blake Eggleston  wrote:
>> 
>> Sorry, I meant trunk, not 4.1 :)
>> 
>>> On Aug 29, 2022, at 1:09 PM, Blake Eggleston  wrote:
>>> 
>>> Hi all, I wanted to propose removing jdk8 support for 4.1. Active support 
>>> ended back in March of this year, and I believe the community has built 
>>> enough confidence in java 11 to make it an uncontroversial change for our 
>>> next major release. Let me know what you think.
>>> 
>>> Thanks,
>>> 
>>> Blake
>> 

Re: [DISCUSS] Removing support for java 8

[Blake Eggleston]

Yes I'd seen the 11+17 thread, but didn't see anything about an explicit jdk8 
removal (ie: removal from CI etc). Ekaterina informed me there was an earlier 
thread covering that though

> On Aug 29, 2022, at 1:09 PM, Blake Eggleston  wrote:
> 
> Hi all, I wanted to propose removing jdk8 support for 4.1. Active support 
> ended back in March of this year, and I believe the community has built 
> enough confidence in java 11 to make it an uncontroversial change for our 
> next major release. Let me know what you think.
> 
> Thanks,
> 
> Blake

Re: [DISCUSS] Removing support for java 8

[Blake Eggleston]

Sorry, I meant trunk, not 4.1 :)

> On Aug 29, 2022, at 1:09 PM, Blake Eggleston  wrote:
> 
> Hi all, I wanted to propose removing jdk8 support for 4.1. Active support 
> ended back in March of this year, and I believe the community has built 
> enough confidence in java 11 to make it an uncontroversial change for our 
> next major release. Let me know what you think.
> 
> Thanks,
> 
> Blake

[DISCUSS] Removing support for java 8

[Blake Eggleston]

Hi all, I wanted to propose removing jdk8 support for 4.1. Active support ended 
back in March of this year, and I believe the community has built enough 
confidence in java 11 to make it an uncontroversial change for our next major 
release. Let me know what you think.

Thanks,

Blake

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

I think we’ve converged on a starting syntax. Are there any additional comments 
before I open a JIRA?

> On Jun 16, 2022, at 10:33 AM, Blake Eggleston  wrote:
> 
> I think in any scenario where the same cell is updated multiple times, the 
> last one would win. The final result for s3 in your example would be 2
> 
>> On Jun 16, 2022, at 10:31 AM, Jon Meredith > <mailto:jmeredit...@gmail.com>> wrote:
>> 
>> The reason I brought up static columns was for cases where multiple 
>> statements update them and there could be ambiguity.
>> 
>> CREATE TABLE tbl
>> {
>>   pk1 int,
>>   ck2 int,
>>   s3 static int,
>>   r4 static int,
>>   PRIMARY KEY (pk1, ck2)
>> }
>> 
>> BEGIN TRANSACTION
>> UPDATE tbl SET s3=1, r4=1 WHERE pk1=1 AND ck2=1;
>> UPDATE tbl SET s3=2, r4=2 WHERE pk1=1 AND ck2=2;
>> COMMIT TRANSACTION
>> 
>> What should the final value be for s3?
>> 
>> This makes me realize I don't understand how upsert statements that touch 
>> the same row would be applied in general within a transaction.
>> If the plan is for only-once-per-row within a transaction, then I think 
>> regular columns and static columns should be split into their own UPSERT 
>> statements.
>> 
>> On Thu, Jun 16, 2022 at 10:40 AM Benedict Elliott Smith > <mailto:bened...@apache.org>> wrote:
>> I like Postgres' approach of letting you declare an exceptional condition 
>> and failing if there is not precisely one result (though I would prefer to 
>> differentiate between 0 row->Null and 2 rows->first row), but once you 
>> permit coercing to NULL I think you have to then treat it like NULL and 
>> permit arithmetic (that itself yields NULL)
>> 
>> This is explicitly stipulated in ANSI SQL 92, in 6.12 > expression>:
>> 
>> General Rules
>> 
>>  1) If the value of any  simply contained in a
>>  is the null value, then the result of
>> the  is the null value.
>> 
>> 
>> On 2022/06/16 16:02:33 Blake Eggleston wrote:
>> > Yeah I'd say NULL is fine for condition evaluation. Reference assignment 
>> > is a little trickier. Assigning null to a column seems ok, but we should 
>> > raise an exception if they're doing math or something that expects a 
>> > non-null value
>> > 
>> > > On Jun 16, 2022, at 8:46 AM, Benedict Elliott Smith > > > <mailto:bened...@apache.org>> wrote:
>> > > 
>> > > AFAICT that standard addresses server-side cursors, not the assignment 
>> > > of a query result to a variable. Could you point to where it addresses 
>> > > variable assignment?
>> > > 
>> > > Postgres has a similar concept, SELECT INTO[1], and it explicitly 
>> > > returns NULL if there are no result rows, unless STRICT is specified in 
>> > > which case an error is returned. My recollection is that T-SQL is also 
>> > > fine with coercing no results to NULL when assigning to a variable or 
>> > > using it in a sub-expression.
>> > > 
>> > > I'm in favour of expanding our functionality here, but I do not see 
>> > > anything fundamentally problematic about the proposal as it stands.
>> > > 
>> > > [1] 
>> > > https://www.postgresql.org/docs/current/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW
>> > >  
>> > > <https://www.postgresql.org/docs/current/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW>
>> > > 
>> > > 
>> > > 
>> > > On 2022/06/13 14:52:41 Konstantin Osipov wrote:
>> > >> * bened...@apache.org <mailto:bened...@apache.org> > > >> <mailto:bened...@apache.org>> [22/06/13 17:37]:
>> > >>> I believe that is a MySQL specific concept. This is one problem with 
>> > >>> mimicking SQL – it’s not one thing!
>> > >>> 
>> > >>> In T-SQL, a Boolean expression is TRUE, FALSE or UNKNOWN[1], and a 
>> > >>> NULL value submitted to a Boolean operator yields UNKNOWN.
>> > >>> 
>> > >>> IF (X) THEN Y does not run Y if X is UNKNOWN;
>> > >>> IF (X) THEN Y ELSE Z does run Z if X is UNKNOWN.
>> > >>> 
>> > >>> So, I think we have evidence that it is fine to interpret NULL
>> > >>> as “false” for the evaluation of IF conditions.
>> > >> 
>> > >> NOT FOUND handler is in ISO/IEC 9075-4:2003 13.2 
>> > >> 
>> > >> In Cassandra results, there is no way to distinguish null values
>> > >> from absence of a row. Branching, thus, without being able to
>> > >> branch based on the absence of a row, whatever specific syntax
>> > >> is used for such branching, is incomplete. 
>> > >> 
>> > >> More broadly, SQL/PSM has exception and condition statements, not
>> > >> just IF statements.
>> > >> 
>> > >> -- 
>> > >> Konstantin Osipov, Moscow, Russia
>> > >> 
>> > 
>> > 
> 

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

I think in any scenario where the same cell is updated multiple times, the last 
one would win. The final result for s3 in your example would be 2

> On Jun 16, 2022, at 10:31 AM, Jon Meredith  wrote:
> 
> The reason I brought up static columns was for cases where multiple 
> statements update them and there could be ambiguity.
> 
> CREATE TABLE tbl
> {
>   pk1 int,
>   ck2 int,
>   s3 static int,
>   r4 static int,
>   PRIMARY KEY (pk1, ck2)
> }
> 
> BEGIN TRANSACTION
> UPDATE tbl SET s3=1, r4=1 WHERE pk1=1 AND ck2=1;
> UPDATE tbl SET s3=2, r4=2 WHERE pk1=1 AND ck2=2;
> COMMIT TRANSACTION
> 
> What should the final value be for s3?
> 
> This makes me realize I don't understand how upsert statements that touch the 
> same row would be applied in general within a transaction.
> If the plan is for only-once-per-row within a transaction, then I think 
> regular columns and static columns should be split into their own UPSERT 
> statements.
> 
> On Thu, Jun 16, 2022 at 10:40 AM Benedict Elliott Smith  <mailto:bened...@apache.org>> wrote:
> I like Postgres' approach of letting you declare an exceptional condition and 
> failing if there is not precisely one result (though I would prefer to 
> differentiate between 0 row->Null and 2 rows->first row), but once you permit 
> coercing to NULL I think you have to then treat it like NULL and permit 
> arithmetic (that itself yields NULL)
> 
> This is explicitly stipulated in ANSI SQL 92, in 6.12  expression>:
> 
> General Rules
> 
>  1) If the value of any  simply contained in a
>  is the null value, then the result of
> the  is the null value.
> 
> 
> On 2022/06/16 16:02:33 Blake Eggleston wrote:
> > Yeah I'd say NULL is fine for condition evaluation. Reference assignment is 
> > a little trickier. Assigning null to a column seems ok, but we should raise 
> > an exception if they're doing math or something that expects a non-null 
> > value
> > 
> > > On Jun 16, 2022, at 8:46 AM, Benedict Elliott Smith  > > <mailto:bened...@apache.org>> wrote:
> > > 
> > > AFAICT that standard addresses server-side cursors, not the assignment of 
> > > a query result to a variable. Could you point to where it addresses 
> > > variable assignment?
> > > 
> > > Postgres has a similar concept, SELECT INTO[1], and it explicitly returns 
> > > NULL if there are no result rows, unless STRICT is specified in which 
> > > case an error is returned. My recollection is that T-SQL is also fine 
> > > with coercing no results to NULL when assigning to a variable or using it 
> > > in a sub-expression.
> > > 
> > > I'm in favour of expanding our functionality here, but I do not see 
> > > anything fundamentally problematic about the proposal as it stands.
> > > 
> > > [1] 
> > > https://www.postgresql.org/docs/current/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW
> > >  
> > > <https://www.postgresql.org/docs/current/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW>
> > > 
> > > 
> > > 
> > > On 2022/06/13 14:52:41 Konstantin Osipov wrote:
> > >> * bened...@apache.org <mailto:bened...@apache.org>  > >> <mailto:bened...@apache.org>> [22/06/13 17:37]:
> > >>> I believe that is a MySQL specific concept. This is one problem with 
> > >>> mimicking SQL – it’s not one thing!
> > >>> 
> > >>> In T-SQL, a Boolean expression is TRUE, FALSE or UNKNOWN[1], and a NULL 
> > >>> value submitted to a Boolean operator yields UNKNOWN.
> > >>> 
> > >>> IF (X) THEN Y does not run Y if X is UNKNOWN;
> > >>> IF (X) THEN Y ELSE Z does run Z if X is UNKNOWN.
> > >>> 
> > >>> So, I think we have evidence that it is fine to interpret NULL
> > >>> as “false” for the evaluation of IF conditions.
> > >> 
> > >> NOT FOUND handler is in ISO/IEC 9075-4:2003 13.2 
> > >> 
> > >> In Cassandra results, there is no way to distinguish null values
> > >> from absence of a row. Branching, thus, without being able to
> > >> branch based on the absence of a row, whatever specific syntax
> > >> is used for such branching, is incomplete. 
> > >> 
> > >> More broadly, SQL/PSM has exception and condition statements, not
> > >> just IF statements.
> > >> 
> > >> -- 
> > >> Konstantin Osipov, Moscow, Russia
> > >> 
> > 
> > 

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

Yeah I'd say NULL is fine for condition evaluation. Reference assignment is a 
little trickier. Assigning null to a column seems ok, but we should raise an 
exception if they're doing math or something that expects a non-null value

> On Jun 16, 2022, at 8:46 AM, Benedict Elliott Smith  
> wrote:
> 
> AFAICT that standard addresses server-side cursors, not the assignment of a 
> query result to a variable. Could you point to where it addresses variable 
> assignment?
> 
> Postgres has a similar concept, SELECT INTO[1], and it explicitly returns 
> NULL if there are no result rows, unless STRICT is specified in which case an 
> error is returned. My recollection is that T-SQL is also fine with coercing 
> no results to NULL when assigning to a variable or using it in a 
> sub-expression.
> 
> I'm in favour of expanding our functionality here, but I do not see anything 
> fundamentally problematic about the proposal as it stands.
> 
> [1] 
> https://www.postgresql.org/docs/current/plpgsql-statements.html#PLPGSQL-STATEMENTS-SQL-ONEROW
> 
> 
> 
> On 2022/06/13 14:52:41 Konstantin Osipov wrote:
>> * bened...@apache.org  [22/06/13 17:37]:
>>> I believe that is a MySQL specific concept. This is one problem with 
>>> mimicking SQL – it’s not one thing!
>>> 
>>> In T-SQL, a Boolean expression is TRUE, FALSE or UNKNOWN[1], and a NULL 
>>> value submitted to a Boolean operator yields UNKNOWN.
>>> 
>>> IF (X) THEN Y does not run Y if X is UNKNOWN;
>>> IF (X) THEN Y ELSE Z does run Z if X is UNKNOWN.
>>> 
>>> So, I think we have evidence that it is fine to interpret NULL
>>> as “false” for the evaluation of IF conditions.
>> 
>> NOT FOUND handler is in ISO/IEC 9075-4:2003 13.2 
>> 
>> In Cassandra results, there is no way to distinguish null values
>> from absence of a row. Branching, thus, without being able to
>> branch based on the absence of a row, whatever specific syntax
>> is used for such branching, is incomplete. 
>> 
>> More broadly, SQL/PSM has exception and condition statements, not
>> just IF statements.
>> 
>> -- 
>> Konstantin Osipov, Moscow, Russia
>> 

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

I see what you mean. We have the EXISTS/NOT EXISTS condition for explicitly 
checking for the existence of a row. One thing the old syntax did is how it 
handled references to columns that don't exist. Previously, if any column 
reference didn't resolve, the update wouldn't apply. With the new syntax, if we 
want to be able to use multiple branches, that's going to be more difficult, 
since taking the 'ELSE' path may not make sense from an application 
perspective. So returning an exception in that case might be the right thing to 
do

> On Jun 15, 2022, at 2:18 PM, Konstantin Osipov  
> wrote:
> 
> * bened...@apache.org  [22/06/16 00:00]:
>> First some history: static rows are an efficiency sop to those
>> who migrated from the historical wide row world, where you could
>> have “global” partition state fetched with every query, and to
>> support the deprecation of thrift and its horrible data model
>> something needed to give – static rows were the result.
>> 
>> However, is the concept generally consistent? I think so. At
>> least, your example seem fine to me, and I can’t see how they
>> violate the “relational model” (whatever that may be). If it
>> helps, you can think of the static columns actually creating a
>> second table, so that you now have two separate tables with the
>> same partition key. These tables are implicitly related via a
>> “full outer join” on the partition key, and you can imagine that
>> you are generally querying a view of this relation.
> 
> This is a model I haven't pondered yet. 
> 
>> In this case, you would expect the outcome you see AFAICT. If
>> you have no restriction on the results, and you have no regular
>> rows and one static row, you would see a single static row
>> result with null regular columns (and a count of 1 row). If you
>> imposed a restriction on regular columns, you would not see the
>> static column as the null regular columns would not match the
>> condition.
>> 
>>> In LWT, a static row appears to exist when there is no regular row matching 
>>> WHERE
>> 
>> I assume you mean the IF clause matches against a static row if
>> you UPDATE tbl SET v = a WHERE p = b IF s = c. This could be an
>> inconsistency, but I think it is not. Recall, UPDATE in CQL is
>> not UPDATE in SQL. SQL would do nothing if the row doesn’t
>> exist, whatever the IF clause might say. CQL is really
>> performing UPSERT.
>> 
>> So, what happens when the WHERE clause doesn’t match a primary
>> key with UPSERT? A row is created. In this case, if you consider
>> that this empty nascent row is used to join with the static
>> “table” for evaluating the IF condition, to decide what you
>> UPSERT, then it all makes sense – to me, anyway.
>> 
>>> NULLs are first-class values, distinguishable from unset values
>> 
>> Could you give an example?
> 
> In SQL, if you FETCH into a VARIABLE and there is no matching row, 
> it won't quietly fill your variable with NULLs or a static cells,
> and leave you wondering what to do next. FETCH will RAISE NOT
> FOUND condition, a kind of exception you can handle separately. 
> Totally different in Cassandra where NULL is a deletion marker and
> NULLs are indistinguishable from missing values.
> 
> -- 
> Konstantin Osipov, Moscow, Russia

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

Regarding modeling syntax after SQL... that approach has pros and cons. 
Supporting an SQL like syntax implies capabilities that we can’t provide, so 
you’re delivering something that looks familiar, but behaves differently, which 
doesn’t help us with usability.

I prefer an approach that supports an accurate mental model of what’s happening 
behind the scenes. I think that should be a design priority for the syntax. 
We’ll be able to build things on top of accord, but the core multi-key cas 
operation isn’t going to change too much.

So I have 2 contrarian proposals:

1. Remove named updates, column references must come from selects. More 
verbose, but crystal clear with regards to when/where values come from.
2. Don’t call these transactions, the term implies things accord doesn’t do. 
Maybe call them CAS BATCH, and terminate them with APPLY or APPLY IF.

Although less exciting, this would simplify the initial implementation, and let 
feature requests and first hand experience inform where and how the syntax 
develops from there.

Blake

> On Jun 13, 2022, at 12:14 PM, Blake Eggleston  wrote:
> 
> Does the IF <...> ABORT simplify reasoning though? If you restrict it to only 
> dealing with the most recent row it would, but referencing the name implies 
> you’d be able to include references from other operations, in which case 
> you’d have the same problem.
> 
> > return instead an exception if the transaction is aborted
> 
> Since the txn is not actually interactive, I think it would be better to 
> receive values instead of an excetion, to understand why the operation was 
> rolled back.
> 
>> On Jun 13, 2022, at 10:32 AM, Aaron Ploetz > <mailto:aaronplo...@gmail.com>> wrote:
>> 
>> Benedict,
>> 
>> I'm really excited about this feature.  I've been observing this 
>> conversation for a while now, and I"m happy to add some thoughts.
>> 
>> We must balance the fact we cannot afford to do everything (yet), against 
>> the need to make sure what we do is reasonably intuitive (to both CQL and 
>> SQL users) and consistent – including with whatever we do in future.
>> 
>> I think taking small steps forward, to build a few complete features as 
>> close to SQL as possible is a good approach.
>> 
>> question we are currently asking: do we want to have a more LWT-like 
>> approach... or do we want a more SQL-like approach
>>  
>> For years now we've been fighting this notion that Cassandra is difficult to 
>> use.  Coming up with specialized syntax isn't going to bridge that divide.  
>> From a (new?) user perspective, the best plan is to stay as consistent with 
>> SQL as possible.
>> 
>> I believe that is a MySQL specific concept. This is one problem with 
>> mimicking SQL – it’s not one thing!
>> 
>> Right?!?!  As if this needed to be more complex.
>> 
>> I think we have evidence that it is fine to interpret NULL as “false” for 
>> the evaluation of IF conditions.
>> 
>> 
>> Agree.  Null == false isn't too much of a leap.
>> 
>> Thanks for taking up the charge on this one.  Glad to see it moving forward!
>> 
>> Thanks,
>> 
>> Aaron
>> 
>> 
>> 
>> On Sun, Jun 12, 2022 at 10:33 AM bened...@apache.org 
>> <mailto:bened...@apache.org> > <mailto:bened...@apache.org>> wrote:
>> Welcome Li, and thanks for your input
>> 
>>  
>> 
>> > When I first saw the syntax, I took it for granted that the condition was 
>> > evaluated against the state AFTER the updates
>> 
>>  
>> 
>> Depending what you mean, I think this is one of the options being 
>> considered. At least, it seems this syntax is most likely to be evaluated 
>> against the values written by preceding statements in the batch, but not the 
>> statement itself (or later ones), as this could lead to nonsensical 
>> statements like
>> 
>>  
>> 
>> BEGIN TRANSACTION
>> 
>> UPDATE tbl SET v = 1 WHERE key = 1 AS tbl
>> 
>> COMMIT TRANSACTION IF tbl.v = 0
>> 
>>  
>> 
>> Where y is never 0 afterwards, so this never succeeds. I take it in this 
>> simple case you would expect the condition to be evaluated against the state 
>> prior to the statement (i.e. the initial state)?
>> 
>>  
>> 
>> But we have a blank slate, so every option is available to us! We just need 
>> to make sure it makes sense to the user, even in uncommon cases.
>> 
>>  
>> 
>> > The IF (Boolean expr) ABORT TRANSACTION would suffer less because users 
>> > may tend to put the condition closer to the related SELECT statem

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

it, and how do we let you 
> express conditions?
> 
>  
> We must balance the fact we cannot afford to do everything (yet), against the 
> need to make sure what we do is reasonably intuitive (to both CQL and SQL 
> users) and consistent – including with whatever we do in future.
> 
>  
> Right now, we have the issue that read-your-writes introduces some complexity 
> to the semantics, particularly around the conditions of execution.
> 
>  
> LWTs impose conditions on the state of all records prior to execution, but 
> their API has a lot of shortcomings. The proposal of COMMIT IF (Boolean expr) 
> is most consistent with this approach. This can be confusing, though, if the 
> condition is evaluated on a value that has been updated by a prior statement 
> in the batch – what value does this global condition get evaluated against?*
> 
>  
> SQL has no such concept, but also SQL is designed to be interactive. 
> Depending on the dialect there’s probably a lot of ways to do this 
> non-interactively in SQL, but we probably cannot cheaply replicate the 
> functionality exactly as we do not (yet) support interactive transactions 
> that they were designed for. To submit a conditional non-interactive 
> transaction in SQL, you would likely use
> 
>  
> IF (X) THEN
> 
> ROLLBACK
> 
> RETURN (ERRCODE)
> 
> END IF
> 
>  
> or
> 
>  
> IF (X) THEN RAISERROR
> 
>  
> So, that is in essence the question we are currently asking: do we want to 
> have a more LWT-like approach (and if so, how do we address this complexity 
> for the user), or do we want a more SQL-like approach (and if so, how do we 
> modify it to make non-interactive transactions convenient, and implementation 
> tractable)
> 
>  
> * This is anyway a shortcoming of existing batches, I think? So it might be 
> we can sweep it under the rug, but I think it will be more relevant here as 
> people execute more complex transactions, and we should ideally have 
> semantics that will work well into the future – including if we later 
> introduce interactive transactions.
> 
>  
>  
>  
>  
>  
> From: Patrick McFadin mailto:pmcfa...@gmail.com>>
> Date: Saturday, 11 June 2022 at 15:33
> To: dev mailto:dev@cassandra.apache.org>>
> Subject: Re: CEP-15 multi key transaction syntax
> 
> I think the syntax is evolving into something pretty complicated, which may 
> be warranted but I wanted to take a step back and be a bit more reflective on 
> what we are trying to accomplish.
> 
>  
> For context, my questions earlier were based on my 20+ years of using SQL 
> transactions across different systems. That's my personal bias when I see the 
> word "database transaction" in this case. When you start a SQL transaction, 
> you are creating a branch of your data that you can operate with until you 
> reach your desired state and then merge it back with a commit. Or if you 
> don't like what you see, use a rollback and act like it never happened. That 
> was the thinking when I asked about interactive sessions. If you are using a 
> driver, that all happens in a batch. I realize that is out of scope here, but 
> that's probably knowledge that is pre-installed in the majority of the user 
> community. 
> 
>  
> Getting to the point, which is developer experience. I'm seeing a 
> philosophical fork in the road which hopefully will generate some comments in 
> the larger user community.
> 
>  
> Path 1) 
> 
> Mimic what's already been available in the SQL community, using existing CQL 
> syntax. (SQL Example using JDBC: 
> https://www.baeldung.com/java-jdbc-auto-commit 
> <https://www.baeldung.com/java-jdbc-auto-commit>)
> 
>  
> Path 2) 
> 
> Chart a new direction with new syntax
> 
>  
> I genuinely don't have a clear answer, but I would love hearing from people 
> on what they think.
> 
>  
> Patrick
> 
>  
> On Fri, Jun 10, 2022 at 12:07 PM bened...@apache.org 
> <mailto:bened...@apache.org>  <mailto:bened...@apache.org>> wrote:
> 
> This might also permit us to remove one result set (the success/failure one) 
> and return instead an exception if the transaction is aborted. This is also 
> more consistent with SQL, if memory serves. That might conflict with 
> returning the other result sets in the event of abort (though that’s up to us 
> ultimately), but it feels like a nicer API for the user – depending on how 
> these exceptions are surfaced in client APIs.
> 
>  
> From: bened...@apache.org <mailto:bened...@apache.org>  <mailto:bened...@apache.org>>
> Date: Friday, 10 June 2022 at 19:59
> To: dev@cassandra.apache.org <mailto:dev@cassandra.

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

Yeah I think that’s intuitive enough. I had been thinking about multiple 
condition branches, but was thinking about something closer to 

IF select.column=5
  UPDATE ... SET ... WHERE key=1;
ELSE IF select.column=6
  UPDATE ... SET ... WHERE key=2;
ELSE
  UPDATE ... SET ... WHERE key=3;
ENDIF
COMMIT TRANSACTION;

Which would make the proposed COMMIT IF we're talking about now a shorthand. Of 
course this would be follow on work.

> On Jun 8, 2022, at 1:20 PM, bened...@apache.org wrote:
> 
> I imagine that conditions would be evaluated against the state prior to the 
> execution of statement against which it is being evaluated, but after the 
> prior statements. I think that should be OK to reason about.
>  
> i.e. we might have a contrived example like:
>  
> BEGIN TRANSACTION
> UPDATE tbl SET a = 1 WHERE k = 1 AS q1
> UPDATE tbl SET a = q1.a + 1 WHERE k = 1 AS q2
> COMMIT TRANSACTION IF q1.a = 0 AND q2.a = 1
>  
> So q1 would read a = 0, but q2 would read a = 1 and set a = 2.
>  
> I think this is probably adequately intuitive? It is a bit atypical to have 
> conditions that wrap the whole transaction though.
>  
> We have another option, of course, which is to offer IF x ROLLBACK 
> TRANSACTION, which is closer to SQL, which would translate the above to:
>  
> BEGIN TRANSACTION
> SELECT a FROM tbl WHERE k = 1 AS q0
> IF q0.a != 0 ROLLBACK TRANSACTION
> UPDATE tbl SET a = 1 WHERE k = 1 AS q1
> IF q1.a != 1 ROLLBACK TRANSACTION
> UPDATE tbl SET a = q1.a + 1 WHERE k = 1 AS q2
> COMMIT TRANSACTION
>  
> This is less succinct, but might be more familiar to users. We could also 
> eschew the ability to read from UPDATE statements entirely in this scheme, as 
> this would then look very much like SQL.
>  
>  
> From: Blake Eggleston 
> Date: Wednesday, 8 June 2022 at 20:59
> To: dev@cassandra.apache.org 
> Subject: Re: CEP-15 multi key transaction syntax
> 
> > It affects not just RETURNING but also conditions that are evaluated 
> > against the row, and if we in future permit using the values from one 
> > select in a function call / write to another table (which I imagine we 
> > will).
> 
> I hadn’t thought about that... using intermediate or even post update values 
> in condition evaluation or function calls seems like it would make it 
> difficult to understand why a condition is or is not applying. On the other 
> hand, it would powerful, especially when using things like database generated 
> values in queries (auto incrementing integer clustering keys or server 
> generated timeuuids being examples that come to mind). Additionally, if we 
> return these values, I guess that would solve the visibility issues I’m 
> worried about. 
> 
> Agreed intermediate values would be straightforward to calculate though.
> 
> 
> On Jun 6, 2022, at 4:33 PM, bened...@apache.org <mailto:bened...@apache.org> 
> wrote:
>  
> It affects not just RETURNING but also conditions that are evaluated against 
> the row, and if we in future permit using the values from one select in a 
> function call / write to another table (which I imagine we will).
>  
> I think that for it to be intuitive we need it to make sense sequentially, 
> which means either calculating it or restricting what can be stated (or 
> abandoning the syntax).
>  
> If we initially forbade multiple UPDATE/INSERT to the same key, but permitted 
> overlapping DELETE (and as many SELECT as you like) that would perhaps make 
> it simple enough? Require for now that SELECTS go first, then DELETE and then 
> INSERT/UPDATE (or vice versa, depending what we want to make simple)?
>  
> FWIW, I don’t think this is terribly onerous to calculate either, since it’s 
> restricted to single rows we are updating, so we could simply maintain a 
> collections of rows and upsert into them as we process the execution. Most 
> transactions won’t need it, I suspect, so we don’t need to worry about 
> perfect efficiency.
>  
>  
> From: Blake Eggleston mailto:beggles...@apple.com>>
> Date: Tuesday, 7 June 2022 at 00:21
> To: dev@cassandra.apache.org <mailto:dev@cassandra.apache.org> 
> mailto:dev@cassandra.apache.org>>
> Subject: Re: CEP-15 multi key transaction syntax
> 
> That's a good question. I'd lean towards returning the final state of things, 
> although I could understand expecting to see intermediate state. Regarding 
> range tombstones, we could require them to precede any updates like selects, 
> but there's still the question of how to handle multiple updates to the same 
> cell when the user has requested we return the post-update state of the cell.
> 
> 
> 
> On Jun 6, 2022, at 4:00 PM, bened...@apache.org <mailto:bened...@apache.org> 
> w

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

> It affects not just RETURNING but also conditions that are evaluated against 
> the row, and if we in future permit using the values from one select in a 
> function call / write to another table (which I imagine we will).

I hadn’t thought about that... using intermediate or even post update values in 
condition evaluation or function calls seems like it would make it difficult to 
understand why a condition is or is not applying. On the other hand, it would 
powerful, especially when using things like database generated values in 
queries (auto incrementing integer clustering keys or server generated 
timeuuids being examples that come to mind). Additionally, if we return these 
values, I guess that would solve the visibility issues I’m worried about. 

Agreed intermediate values would be straightforward to calculate though.

> On Jun 6, 2022, at 4:33 PM, bened...@apache.org wrote:
> 
> It affects not just RETURNING but also conditions that are evaluated against 
> the row, and if we in future permit using the values from one select in a 
> function call / write to another table (which I imagine we will).
>  
> I think that for it to be intuitive we need it to make sense sequentially, 
> which means either calculating it or restricting what can be stated (or 
> abandoning the syntax).
>  
> If we initially forbade multiple UPDATE/INSERT to the same key, but permitted 
> overlapping DELETE (and as many SELECT as you like) that would perhaps make 
> it simple enough? Require for now that SELECTS go first, then DELETE and then 
> INSERT/UPDATE (or vice versa, depending what we want to make simple)?
>  
> FWIW, I don’t think this is terribly onerous to calculate either, since it’s 
> restricted to single rows we are updating, so we could simply maintain a 
> collections of rows and upsert into them as we process the execution. Most 
> transactions won’t need it, I suspect, so we don’t need to worry about 
> perfect efficiency.
>  
>  
> From: Blake Eggleston 
> Date: Tuesday, 7 June 2022 at 00:21
> To: dev@cassandra.apache.org 
> Subject: Re: CEP-15 multi key transaction syntax
> 
> That's a good question. I'd lean towards returning the final state of things, 
> although I could understand expecting to see intermediate state. Regarding 
> range tombstones, we could require them to precede any updates like selects, 
> but there's still the question of how to handle multiple updates to the same 
> cell when the user has requested we return the post-update state of the cell.
> 
> 
> On Jun 6, 2022, at 4:00 PM, bened...@apache.org <mailto:bened...@apache.org> 
> wrote:
>  
> > if multiple updates end up touching the same cell, I’d expect the last one 
> > to win
>  
> Hmm, yes I suppose range tombstones are a plausible and reasonable thing to 
> mix with inserts over the same key range.
>  
> What’s your present thinking about the idea of handling returning the values 
> as of a given point in the sequential execution then?
>  
> The succinct syntax is I think highly desirable for user experience, but this 
> does complicate it a bit if we want to remain intuitive.
>  
>  
>  
>  
> From: Blake Eggleston mailto:beggles...@apple.com>>
> Date: Monday, 6 June 2022 at 23:17
> To: dev@cassandra.apache.org <mailto:dev@cassandra.apache.org> 
> mailto:dev@cassandra.apache.org>>
> Subject: Re: CEP-15 multi key transaction syntax
> 
> Hi all,
> 
> Thanks for all the input and questions so far. Glad people are excited about 
> this!
> 
> I didn’t have any free time to respond this weekend, although it looks like 
> Benedict has responded to most of the questions so far, so if I don’t respond 
> to a question you asked here, you can interpret that as “what Benedict said” 
> :).
> 
> 
> Jeff, 
> 
> > Is there a new keyword for “partition (not) exists” or is it inferred by 
> > the select?
> 
> I'd intended this to be worked out from the select statement, ie: if the 
> read/reference is null/empty, then it doesn't exist, whether you're 
> interested in the partition, row, or cell. So I don't think we'd need an 
> additional keyword there. I think that would address partition exists / not 
> exists use cases?
> 
> > And would you allow a transaction that had > 1 named select and no 
> > modification statements, but commit if 1=1 ?
> 
> Yes, an unconditional commit (ie: just COMMIT TRANSACTION; without an IF) 
> would be part of the syntax. Also, running a txn that doesn’t contain updates 
> wouldn’t be a problem.
> 
> Patrick, I think Benedict answered your questions? Glad you got the joke :)
> 
> Alex,
> 
> > 1. Dependant SELECTs
> > 2. Dependant UPDATEs
> > 3. UPDATE from seconda

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

That's a good question. I'd lean towards returning the final state of things, 
although I could understand expecting to see intermediate state. Regarding 
range tombstones, we could require them to precede any updates like selects, 
but there's still the question of how to handle multiple updates to the same 
cell when the user has requested we return the post-update state of the cell.

> On Jun 6, 2022, at 4:00 PM, bened...@apache.org wrote:
> 
> > if multiple updates end up touching the same cell, I’d expect the last one 
> > to win
>  
> Hmm, yes I suppose range tombstones are a plausible and reasonable thing to 
> mix with inserts over the same key range.
>  
> What’s your present thinking about the idea of handling returning the values 
> as of a given point in the sequential execution then?
>  
> The succinct syntax is I think highly desirable for user experience, but this 
> does complicate it a bit if we want to remain intuitive.
>  
>  
>  
>  
> From: Blake Eggleston 
> Date: Monday, 6 June 2022 at 23:17
> To: dev@cassandra.apache.org 
> Subject: Re: CEP-15 multi key transaction syntax
> 
> Hi all,
> 
> Thanks for all the input and questions so far. Glad people are excited about 
> this!
> 
> I didn’t have any free time to respond this weekend, although it looks like 
> Benedict has responded to most of the questions so far, so if I don’t respond 
> to a question you asked here, you can interpret that as “what Benedict said” 
> :).
> 
> 
> Jeff, 
> 
> > Is there a new keyword for “partition (not) exists” or is it inferred by 
> > the select?
> 
> I'd intended this to be worked out from the select statement, ie: if the 
> read/reference is null/empty, then it doesn't exist, whether you're 
> interested in the partition, row, or cell. So I don't think we'd need an 
> additional keyword there. I think that would address partition exists / not 
> exists use cases?
> 
> > And would you allow a transaction that had > 1 named select and no 
> > modification statements, but commit if 1=1 ?
> 
> Yes, an unconditional commit (ie: just COMMIT TRANSACTION; without an IF) 
> would be part of the syntax. Also, running a txn that doesn’t contain updates 
> wouldn’t be a problem.
> 
> Patrick, I think Benedict answered your questions? Glad you got the joke :)
> 
> Alex,
> 
> > 1. Dependant SELECTs
> > 2. Dependant UPDATEs
> > 3. UPDATE from secondary index (or SASI)
> > 5. UPDATE with predicate on non-primary key
> 
> The full primary key must be defined as part of the statement, and you can’t 
> use column references to define them, so you wouldn’t be able to run these.
> 
> > MVs
> 
> To prevent being spread too thin, both in syntax design and implementation 
> work, I’d like to limit read and write operations in the initial 
> implementation to vanilla selects, updates, inserts, and deletes. Once we 
> have a solid implementation of multi-key/table transactions supporting 
> foundational operations, we can start figuring out how the more advanced 
> pieces can be best supported. Not a great answer to your question, but a 
> related tangent I should have included in my initial email.
> 
> > ... RETURNING ...
> 
> I like the idea of the returning statement, but to echo what Benedict said, I 
> think any scheme for specifying data to be returned should apply the same to 
> select and update statements, since updates can have underlying reads that 
> the user may be interested in. I’d mentioned having an optional RETURN 
> statement in addition to automatically returning selects in my original email.
> 
> > ... WITH ...
> 
> I like the idea of defining statement names at the beginning of a statement, 
> since I could imagine mapping names to selects might get difficult if there 
> are a lot of columns in the select or update, but beginning each statement 
> with `WITH ` reduces readability imo. Maybe putting the name after the 
> first term of the statement (ie: `SELECT * AS  WHERE...`, `UPDATE table 
> AS  SET ...`, `INSERT INTO table AS  (...) VALUES (...);`) would 
> be improve finding names without harming overall readability?
> 
> Benedict,
> 
> > I agree that SELECT statements should be required to go first.
> 
> +1
> 
> > There only remains the issue of conditions imposed upon 
> > UPDATE/INSERT/DELETE statements when there are multiple statements that 
> > affect the same primary key. I think we can (and should) simply reject such 
> > queries for now, as it doesn’t make much sense to have multiple statements 
> > for the same primary key in the same transaction.
> 
> Unfortunately, I think there are use cases for both multiple select

Re: CEP-15 multi key transaction syntax

[Blake Eggleston]

Hi all,

Thanks for all the input and questions so far. Glad people are excited about 
this!

I didn’t have any free time to respond this weekend, although it looks like 
Benedict has responded to most of the questions so far, so if I don’t respond 
to a question you asked here, you can interpret that as “what Benedict said” :).


Jeff, 

> Is there a new keyword for “partition (not) exists” or is it inferred by the 
> select?

I'd intended this to be worked out from the select statement, ie: if the 
read/reference is null/empty, then it doesn't exist, whether you're interested 
in the partition, row, or cell. So I don't think we'd need an additional 
keyword there. I think that would address partition exists / not exists use 
cases?

> And would you allow a transaction that had > 1 named select and no 
> modification statements, but commit if 1=1 ?

Yes, an unconditional commit (ie: just COMMIT TRANSACTION; without an IF) would 
be part of the syntax. Also, running a txn that doesn’t contain updates 
wouldn’t be a problem.

Patrick, I think Benedict answered your questions? Glad you got the joke :)

Alex,

> 1. Dependant SELECTs
> 2. Dependant UPDATEs
> 3. UPDATE from secondary index (or SASI)
> 5. UPDATE with predicate on non-primary key

The full primary key must be defined as part of the statement, and you can’t 
use column references to define them, so you wouldn’t be able to run these.

> MVs

To prevent being spread too thin, both in syntax design and implementation 
work, I’d like to limit read and write operations in the initial implementation 
to vanilla selects, updates, inserts, and deletes. Once we have a solid 
implementation of multi-key/table transactions supporting foundational 
operations, we can start figuring out how the more advanced pieces can be best 
supported. Not a great answer to your question, but a related tangent I should 
have included in my initial email.

> ... RETURNING ...

I like the idea of the returning statement, but to echo what Benedict said, I 
think any scheme for specifying data to be returned should apply the same to 
select and update statements, since updates can have underlying reads that the 
user may be interested in. I’d mentioned having an optional RETURN statement in 
addition to automatically returning selects in my original email.

> ... WITH ...

I like the idea of defining statement names at the beginning of a statement, 
since I could imagine mapping names to selects might get difficult if there are 
a lot of columns in the select or update, but beginning each statement with 
`WITH ` reduces readability imo. Maybe putting the name after the first 
term of the statement (ie: `SELECT * AS  WHERE...`, `UPDATE table AS 
 SET ...`, `INSERT INTO table AS  (...) VALUES (...);`) would be 
improve finding names without harming overall readability?

Benedict,

> I agree that SELECT statements should be required to go first.

+1

> There only remains the issue of conditions imposed upon UPDATE/INSERT/DELETE 
> statements when there are multiple statements that affect the same primary 
> key. I think we can (and should) simply reject such queries for now, as it 
> doesn’t make much sense to have multiple statements for the same primary key 
> in the same transaction.

Unfortunately, I think there are use cases for both multiple selects and 
updates for the same primary key in a txn. Selects aren’t as problematic, but 
if multiple updates end up touching the same cell, I’d expect the last one to 
win. This would make dealing with range tombstones a little trickier, since the 
default behavior of alternating updates and range tombstones affecting the same 
cells is not intuitive, but I don’t think it would be too bad.


Something that’s come up a few times, and that I’ve also been thinking about is 
whether to return the values that were originally read, or the values written 
with the update to the client, and there are use cases for both. I don’t 
remember who suggested it, but I think returning the original values from named 
select statements, and the post-update values from named update statements is a 
good way to handle both. Also, while returning the contents of the mutation 
would be the easiest, implementation wise, swapping cell values from the 
updates named read would be most useful, since a txn won’t always result in an 
update, in which case we’d just return the select.

Thanks,

Blake



> On Jun 6, 2022, at 9:41 AM, Henrik Ingo  wrote:
> 
> On Mon, Jun 6, 2022 at 5:28 PM bened...@apache.org 
>   > wrote:
> > One way to make it obvious is to require the user to explicitly type the 
> > SELECTs and then to require that all SELECTs appear before 
> > UPDATE/INSERT/DELETE.
> 
>  
> 
> Yes, I agree that SELECT statements should be required to go first.
> 
>  
> 
> However, I think this is sufficient and we can retain the shorter format for 
> RETURNING. There only remains the issue of conditions 

CEP-15 multi key transaction syntax

[Blake Eggleston]

Hi dev@,

I’ve been working on a draft syntax for Accord transactions and wanted to bring 
what I have to the dev list to solicit feedback and build consensus before 
moving forward with it. The proposed transaction syntax is intended to be an 
extended batch syntax. Basically batches with selects, and an optional 
condition at the end. To facilitate conditions against an arbitrary number of 
select statements, you can also name the statements, and reference columns in 
the results. To cut down on the number of operations needed, select values can 
also be used in updates, including some math operations. Parameterization of 
literals is supported the same as other statements.

Here's an example selecting a row from 2 tables, and issuing updates for each 
row if a condition is met:

BEGIN TRANSACTION;
  SELECT * FROM users WHERE name='blake' AS user;
  SELECT * from cars WHERE model='pinto' AS car;
  UPDATE users SET miles_driven = user.miles_driven + 30 WHERE name='blake';
  UPDATE cars SET miles_driven = car.miles_driven + 30 WHERE model='pinto';
COMMIT TRANSACTION IF car.is_running;

This can be simplified by naming the updates with an AS  syntax. If 
updates are named, a corresponding read is generated behind the scenes and its 
values inform the update.

Here's an example, the query is functionally identical to the previous query. 
In the case of the user update, a read is still performed behind the scenes to 
enable the calculation of miles_driven + 30, but doesn't need to be named since 
it's not referenced anywhere else.

BEGIN TRANSACTION;
  UPDATE users SET miles_driven += 30 WHERE name='blake';
  UPDATE cars SET miles_driven += 30 WHERE model='pinto' AS car;
COMMIT TRANSACTION IF car.is_running;

Here’s another example, performing the canonical bank transfer:

BEGIN TRANSACTION;
  UPDATE accounts SET balance += 100 WHERE name='blake' AS blake;
  UPDATE accounts SET balance -= 100 WHERE name='benedict' AS benedict;
COMMIT TRANSACTION IF blake EXISTS AND benedict.balance >= 100;

As you can see from the examples, column values can be referenced via a dot 
syntax, ie: . -> select1.value. Since the read portion of 
the transaction is performed before evaluating conditions or applying updates, 
values read can be freely applied to non-primary key values in updates. Select 
statements used either in checking a condition or creating an update must be 
restricted to a single row, either by specifying the full primary key or a 
limit of 1. Multi-row selects are allowed, but only for returning data to the 
client (see below).

For evaluating conditions, = & != are available for all types, <, <=, >, >= are 
available for numerical types, and EXISTS, NOT EXISTS can be used for 
partitions, rows, and values. If any column references cannot be satisfied by 
the result of the reads, the condition implicitly fails. This prevents having 
to include a bunch of exists statements.

On completion, an operation would return a boolean value indicating the 
operation had been applied, and a result set for each named select (but not 
named update). We could also support an optional RETURN keyword, which would 
allow the user to only return specific named selects (ie: RETURN select1, 
select2).

Let me know what you think!

Blake

Re: [DISCUSS] Mentoring newcomers

[Blake Eggleston]

I’m happy to help out

> On Nov 12, 2021, at 9:04 AM, Benjamin Lerer  wrote:
> 
> Hi everybody
> 
> As discussed in the *Creating a new slack channel for newcomers* thead, a
> solution to help newcomers engage with the project would be to provide a
> list of mentors that newcomers can contact when they feel insecure about
> asking questions through our cassandra-dev channel or through the mailing
> list.
> 
> I would like to collect the list of people that are interested in helping
> out newcomers so that we can post that list on our website.


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-15: General Purpose Transactions

[Blake Eggleston]

1. +1
2. +1
3. +1

> On Oct 14, 2021, at 9:31 AM, bened...@apache.org wrote:
> 
> Hi everyone,
> 
> I would like to start a vote on this CEP, split into three sub-decisions, as 
> discussion has been circular for some time.
> 
> 1. Do you support adopting this CEP?
> 2. Do you support the transaction semantics proposed by the CEP for Cassandra?
> 3. Do you support an incremental approach to developing transactions in 
> Cassandra, leaving scope for future development?
> 
> The first vote is a consensus vote of all committers, the second and third 
> however are about project direction and therefore are simple majority votes 
> of the PMC.
> 
> Recall that all -1 votes must be accompanied by an explanation. If you reject 
> the CEP only on grounds (2) or (3) you should not veto the proposal. If a 
> majority reject grounds (2) or (3) then transaction developments will halt 
> for the time being.
> 
> This vote will be open for 72 hours.


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Tradeoffs for Cassandra transaction management

[Blake Eggleston]

Hi Henrik,

I would agree that the local serial experience for valid use cases should be 
supported in some form before legacy LWT is replaced by Accord.

Regarding your read committed proposal, I think this CEP discussion has already 
spent too much time talking about hypothetical SQL implementations, and I’d 
like to avoid veering off course again. However, since you’ve asked a well 
thought out question with concrete goals and implementation ideas, I’m happy to 
answer it. I just ask that if you want to discuss it beyond my reply, you start 
a separate ‘[IDEA] Read committed transaction with Accord’ thread where we 
could talk about it a bit more without it feeling like we need to delay a vote.

So I think it could work with some modifications. 

First you’d need to perform your select statements as accord reads, not quorum 
reads. Otherwise you may not see writes that have been (or could have been) 
committed. A multi-partition write could also appear to become undone, if a 
write commit has not reached one of the keys or needs to be recovered.

Second, when you talk about transforming mutations, I’m assuming you’re talking 
about confirming primary keys do or do not exist, and supporting 
auto-incrementing primary keys. To confirm primary keys do or do not exist, 
you’d also need to perform an accord read also. For auto-incrementing primary 
keys, you’d need to do an accord read/write operation to increment a counter 
somewhere (or just use uuids).

Finally, read committed does lock rows, so you’d still need to perform a read 
on commit to confirm that the rows being written to haven’t been modified since 
the transaction began.

Thanks,

Blake


> On Oct 12, 2021, at 1:54 PM, Henrik Ingo  wrote:
> 
> Hi all
> 
> I was expecting to stay out of the way while a vote on CEP-15 seemed
> imminent. But discussing this tradeoffs thread with Jonathan, he encouraged
> me to say these points in my own words, so here we are.
> 
> 
> On Sun, Oct 10, 2021 at 7:17 AM Blake Eggleston
> mailto:beggles...@apple.com.invalid>> wrote:
> 
>> 1. Is it worth giving up local latencies to get full global consistency?
>> Most LWT use cases use
>> LOCAL_SERIAL.
>> 
>> This isn’t a tradeoff that needs to be made. There’s nothing about Accord
>> that prevents performing consensus in one DC and replicating the writes to
>> others. That’s not in scope for the initial work, but there’s no reason it
>> couldn’t be handled as a follow on if needed. I agree with Jeff that
>> LOCAL_SERIAL and LWTs are not usually done with a full understanding of the
>> implications, but there are some valid use cases. For instance, you can
>> enable an OLAP service to operate against another DC without impacting the
>> primary, assuming the service can tolerate inconsistency for data written
>> since the last repair, and there are some others.
>> 
>> 
> Let's start with the stated goal that CEP-15 is intended to be a better
> version of LWT.
> 
> Reading all the discussion, I feel like addressing the LOCAL_SERIAL /
> LOCAL_QUORUM use case is the one thing where Accord isn't strictly an
> improvement over LWT. I don't agree that Accord will just be so much faster
> anyway, that it would compensate a single network roundtrip around the
> world. Four LWT round-trips with LOCAL_SERIAL will still only be on the
> order of 10 ms, but global latencies for just a single round trip are
> hundreds of ms.
> 
> So, my suggestion to resolve this discussion would be that "local quorum
> latency experience" should be included in CEP-15 to meet its stated goal.
> If I have understood the CEP process correctly, this merely means that we
> agree this is a valid and significant use case in the Cassandra ecosystem.
> It doesn't mean that everything in the CEP must be released in a single v1
> release. At least personally I don't necessarily need to see a very
> detailed design for the implementation. But I'm optimistic it would resolve
> one open discussion if it was codified in the CEP that this is a use case
> that needs to be addressed.
> 
> 
>> 2. Is it worth giving up the possibility of SQL support, to get the
>> benefits of deterministic transaction design?
>> 
>> This is a false dilemma. Today, we’re proposing a deterministic
>> transaction design that addresses some very common user pain points. SQL
>> addresses different user pain point. If someone wants to add an sql
>> implementation in the future they can a) build it on top of accord b)
>> extend or improve accord or c) implement a separate system. The right
>> choice will depend on their goals, but accord won’t prevent work on it, the
>> same way the original lwt design isn’t preventing work on multi-partition
>> transactions. In the wo

Re: Tradeoffs for Cassandra transaction management

[Blake Eggleston]

Let’s get back on topic.

Jonathan, in your opening email you stated that, in your view, the 2 main areas 
of tradeoff were:

> 1. Is it worth giving up local latencies to get full global consistency? 

Now we’ve established that we don’t need to give up local latencies with 
Accord, which leaves:

> 2. Is it worth giving up the possibility of SQL support, to get the benefits 
> of deterministic transaction design?

I pointed out that this was a false dilemma and that, in the worst case, a 
hypothetical SQL feature could have it’s own consensus system. I hope that 
won’t be necessary, but as I later pointed out (and you did not address, 
although maybe I should have phrased it as a question), if we’re going to weigh 
accord against a hypothetical SQL feature that lacks design goals, or any clear 
ideas about how it might be implemented, how can we rule that out?

So Jonathan, how can we rule that out? How can we have a productive discussion 
about a feature you yourself are unable to describe in any meaningful detail?

> On Oct 11, 2021, at 6:34 PM, Jonathan Ellis  wrote:
> 
> On Mon, Oct 11, 2021 at 5:11 PM bened...@apache.org 
> wrote:
> 
>> If we want to fully unpack this particular point, as far as I can tell
>> claiming ANSI SQL would indeed require interactive transactions in which
>> arbitrary conditional work may be performed by a client within a
>> transaction in response to other actions within that transaction.
>> 
>> However:
>> 
>>  1.  The ANSI SQL standard permits these transactions to fail and
>> rollback (e.g. in the event that your optimistic transaction fails). So if
>> you want to be pedantic, you may modify my statement to “SQL does not
>> necessitate support for abort-free interactive transactions” and we can
>> leave it there.
>> 
>>  2.  I would personally consider “SQL support” to include the capability
>> of defining arbitrary SQL stored procedures that may be executed by clients
>> in an interactive session
> 
> 
> I note your personal preference and I further note that this is not the
> common understanding of "SQL support" in the industry.  If you tell 100
> developers that your database supports SQL, then at least 99 of them are
> going to assume that you can work with APIs like JDBC that expose
> interactive transactions as a central feature, and hence that you will be
> reasonably compatible with the vast array of SQL-based applications out
> there.
> 
> Historical side note: VoltDB tried to convince people that stored
> procedures were good enough.  It didn't work, and VoltDB had to add
> interactive transactions as fast as they could.
> 
>  3.  Most importantly, as I pointed out in the previous email, Accord is
>> compatible with a YugaByte/Cockroach-like approach, and indeed makes this
>> approach both easier to accomplish and enables stronger isolation than the
>> equivalent Raft-based approach. These approaches are able to reduce the
>> number of conflicts, at a cost of significantly higher transaction
>> management burden.
>> 
> 
> If you're saying that you could use Accord instead of Raft or Paxos, and
> layer 2PC on top of that as in Spanner, then I agree, but I don't think
> that is a very good design, as you would no longer get any of the benefits
> of the deterministic approach you started with.  If you mean something
> else, then perhaps an example would help clarify.
> 
> -- 
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Tradeoffs for Cassandra transaction management

[Blake Eggleston]

> Come on Blake, you have all been developing software long enough to know
> that "there's nothing about Accord that prevents this" is close to
> meaningless.
> 
> If it's so easy to address an overwhelmingly popular use case, then let's
> add it to the initial work.


This is moving the goal posts. The concern I was addressing implied this wasn’t 
possible with Accord and asked if we should prefer “a design that allows local 
serialization with EC between regions”. Accord is a design that allows this, 
and support for it is an implementation detail. Whether or not it’s in scope 
for the initial work is a project planning discussion, not a transaction 
management protocol tradeoff discussion.

> I think this is the crux of our disagreement, I very much want to avoid a
> future where we have to maintain two separate consensus systems.


I want to avoid it also, but if we’re going to compare Accord against a 
hypothetical SQL feature that seems to lack design goals, or any clear ideas 
about how it might be implemented, I don’t think we can rule it out.


> On Oct 11, 2021, at 6:02 AM, Jonathan Ellis  wrote:
> 
> On Sat, Oct 9, 2021 at 11:23 PM Blake Eggleston
> mailto:beggles...@apple.com.invalid>> wrote:
> 
>> 1. Is it worth giving up local latencies to get full global consistency?
>> Most LWT use cases use
>> LOCAL_SERIAL.
>> 
>> This isn’t a tradeoff that needs to be made. There’s nothing about Accord
>> that prevents performing consensus in one DC and replicating the writes to
>> others. That’s not in scope for the initial work, but there’s no reason it
>> couldn’t be handled as a follow on if needed. I agree with Jeff that
>> LOCAL_SERIAL and LWTs are not usually done with a full understanding of the
>> implications, but there are some valid use cases. For instance, you can
>> enable an OLAP service to operate against another DC without impacting the
>> primary, assuming the service can tolerate inconsistency for data written
>> since the last repair, and there are some others.
>> 
> 
> Come on Blake, you have all been developing software long enough to know
> that "there's nothing about Accord that prevents this" is close to
> meaningless.
> 
> If it's so easy to address an overwhelmingly popular use case, then let's
> add it to the initial work.
> 
> 2. Is it worth giving up the possibility of SQL support, to get the
>> benefits of deterministic transaction design?
>> 
>> This is a false dilemma. Today, we’re proposing a deterministic
>> transaction design that addresses some very common user pain points. SQL
>> addresses different user pain point. If someone wants to add an sql
>> implementation in the future they can a) build it on top of accord b)
>> extend or improve accord or c) implement a separate system. The right
>> choice will depend on their goals, but accord won’t prevent work on it, the
>> same way the original lwt design isn’t preventing work on multi-partition
>> transactions. In the worst case, if the goals of a hypothetical sql project
>> are different enough to make them incompatible with accord, I don’t see any
>> reason why we couldn’t have 2 separate consensus systems, so long as people
>> are willing to maintain them and the use cases and available technologies
>> justify it.
>> 
> 
> I think this is the crux of our disagreement, I very much want to avoid a
> future where we have to maintain two separate consensus systems.

Re: Tradeoffs for Cassandra transaction management

[Blake Eggleston]

1. Is it worth giving up local latencies to get full global consistency? Most 
LWT use cases use
LOCAL_SERIAL.

This isn’t a tradeoff that needs to be made. There’s nothing about Accord that 
prevents performing consensus in one DC and replicating the writes to others. 
That’s not in scope for the initial work, but there’s no reason it couldn’t be 
handled as a follow on if needed. I agree with Jeff that LOCAL_SERIAL and LWTs 
are not usually done with a full understanding of the implications, but there 
are some valid use cases. For instance, you can enable an OLAP service to 
operate against another DC without impacting the primary, assuming the service 
can tolerate inconsistency for data written since the last repair, and there 
are some others.

2. Is it worth giving up the possibility of SQL support, to get the benefits of 
deterministic transaction design? 

This is a false dilemma. Today, we’re proposing a deterministic transaction 
design that addresses some very common user pain points. SQL addresses 
different user pain point. If someone wants to add an sql implementation in the 
future they can a) build it on top of accord b) extend or improve accord or c) 
implement a separate system. The right choice will depend on their goals, but 
accord won’t prevent work on it, the same way the original lwt design isn’t 
preventing work on multi-partition transactions. In the worst case, if the 
goals of a hypothetical sql project are different enough to make them 
incompatible with accord, I don’t see any reason why we couldn’t have 2 
separate consensus systems, so long as people are willing to maintain them and 
the use cases and available technologies justify it.

-Blake

> On Oct 9, 2021, at 9:54 AM, Jonathan Ellis  wrote:
> 
> * Hi all,After calling several times for a broader discussion of goals and
> tradeoffs around transaction management in the CEP-15 thread, I’ve put
> together a short analysis to kick that off.Here is a table that summarizes
> the state of the art for distributed transactions that offer
> serializability, i.e., a superset of what you can get with LWT.  (The most
> interesting option that this eliminates is RAMP.)Since I'm not sure how
> this will render outside gmail, I've also uploaded it here:
> https://imgur.com/a/SCZ8jex
> SpannerCockroachCalvin/FaunaSLOG (see
> below)Write latencyGlobal Paxos, plus 2pc for multi-partition.For
> intercontinental replication this is 100+ms.  Cloud Spanner does not allow
> truly global deployments for this reason.Single-region Paxos, plus 2pc.
> I’m not very clear on how this works but it results in non-strict
> serializability.I didn’t find actual numbers for CR other than “2ms in a
> single AZ” which is not a typical scenario.Global Raft.  Fauna posts actual
> numbers of ~70ms in production which I assume corresponds to a multi-region
> deployment with all regions in the USA.  SLOG paper says true global Calvin
> is 200+ms.Single-region Paxos (common case) with fallback to multi-region
> Paxos.Under 10ms.Scalability bottlenecksLocks held during cross-region
> replicationSame as SpannerOLLP approach required when PKs are not known in
> advance (mostly for indexed queries) -- results in retries under
> contentionSame as CalvinRead latency at serial consistencyTimestamp from
> Paxos leader (may be cross-region), then read from local replica.Same as
> Spanner, I thinkSame as writesSame as writesMaximum serializability
> flavorStrictUn-strictStrictStrictSupport for other isolation
> levels?SnapshotNoSnapshot (in Fauna)Paper mentions dropping from
> strict-serializable to only serializable.  Probably could also support
> Snapshot like Fauna.Interactive transaction support (req’d for
> SQL)YesYesNoNoPotential for grafting onto C*NightmareNightmareReasonable,
> Calvin is relatively simple and the storage assumptions it makes are
> minimalI haven’t thought about this enough. SLOG may require versioned
> storage, e.g. see this comment
> .(I
> have not included Accord here because it’s not sufficiently clear to me how
> to create a full transaction manager from the Accord protocol, so I can’t
> analyze many of the properties such a system would have.  The most obvious
> solution would be “Calvin but with Accord instead of Raft”, but since
> Accord already does some Calvin-like things that seems like it would result
> in some suboptimal redundancy.)After putting the above together it seems to
> me that the two main areas of tradeoff are, 1. Is it worth giving up local
> latencies to get full global consistency?  Most LWT use cases use
> LOCAL_SERIAL.  While all of the above have more efficient designs than LWT,
> it’s still true that global serialization will require 100+ms in the
> general case due to physical transmission latency.  So a design that allows
> local serialization with EC 

Re: [DISCUSS] CEP-15: General Purpose Transactions

[Blake Eggleston]

You could establish a lower timestamp bound and buffer transaction state on the 
coordinator, then make the commit an operation that only applies if all 
partitions involved haven’t been changed by a more recent timestamp. You could 
also implement mvcc either in the storage layer or for some period of time by 
buffering commits on each replica before applying.

> On Sep 29, 2021, at 6:18 PM, Jonathan Ellis  wrote:
> 
> How are interactive transactions possible with Accord?
> 
> 
> 
> On Tue, Sep 21, 2021 at 11:56 PM bened...@apache.org 
> wrote:
> 
>> Could you explain why you believe this trade-off is necessary? We can
>> support full SQL just fine with Accord, and I hope that we eventually do so.
>> 
>> This domain is incredibly complex, so it is easy to reach wrong
>> conclusions. I would invite you again to propose a system for discussion
>> that you think offers something Accord is unable to, and that you consider
>> desirable, and we can work from there.
>> 
>> To pre-empt some possible discussions, I am not aware of anything we
>> cannot do with Accord that we could do with either Calvin or Spanner.
>> Interactive transactions are possible on top of Accord, as are transactions
>> with an unknown read/write set. In each case the only cost is that they
>> would use optimistic concurrency control, which is no worse the spanner
>> derivatives anyway (which I have to assume is your benchmark in this
>> regard). I do not expect to deliver either functionality initially, but
>> Accord takes us most of the way there for both.
>> 
>> 
>> From: Jonathan Ellis 
>> Date: Wednesday, 22 September 2021 at 05:36
>> To: dev 
>> Subject: Re: [DISCUSS] CEP-15: General Purpose Transactions
>> Right, I'm looking for exactly a discussion on the high level goals.
>> Instead of saying "here's the goals and we ruled out X because Y" we should
>> start with a discussion around, "Approach A allows X and W, approach B
>> allows Y and Z" and decide together what the goals should be and and what
>> we are willing to trade to get those goals, e.g., are we willing to give up
>> global strict serializability to get the ability to support full SQL.  Both
>> of these are nice to have!
>> 
>> On Tue, Sep 21, 2021 at 9:52 PM bened...@apache.org 
>> wrote:
>> 
>>> Hi Jonathan,
>>> 
>>> These other systems are incompatible with the goals of the CEP. I do
>>> discuss them (besides 2PC) in both the whitepaper and the CEP, and will
>>> summarise that discussion below. A true and accurate comparison of these
>>> other systems is essentially intractable, as there are complex subtleties
>>> to each flavour, and those who are interested would be better served by
>>> performing their own research.
>>> 
>>> I think it is more productive to focus on what we want to achieve as a
>>> community. If you believe the goals of this CEP are wrong for the
>> project,
>>> let’s focus on that. If you want to compare and contrast specific facets
>> of
>>> alternative systems that you consider to be preferable in some dimension,
>>> let’s do that here or in a Q as proposed by Joey.
>>> 
>>> The relevant goals are that we:
>>> 
>>> 
>>>  1.  Guarantee strict serializable isolation on commodity hardware
>>>  2.  Scale to any cluster size
>>>  3.  Achieve optimal latency
>>> 
>>> The approach taken by Spanner derivatives is rejected by (1) because they
>>> guarantee only Serializable isolation (they additionally fail (3)). From
>>> watching talks by YugaByte, and inferring from Cockroach’s
>>> panic-cluster-death under clock skew, this is clearly considered by
>>> everyone to be undesirable but necessary to achieve scalability.
>>> 
>>> The approach taken by FaunaDB (Calvin) is rejected by (2) because its
>>> sequencing layer requires a global leader process for the cluster, which
>> is
>>> incompatible with Cassandra’s scalability requirements. It additionally
>>> fails (3) for global clients.
>>> 
>>> Two phase commit fails (3). As an aside, AFAICT DynamoDB is today a
>>> Spanner clone for its multi-key transaction functionality, not 2PC.
>>> 
>>> Systems such as RAMP with even weaker isolation are not considered for
>> the
>>> simple reason that they do not even claim to meet (1).
>>> 
>>> If we want to additionally offer weaker isolation levels than
>>> Serializable, such as that provided by the recent RAMP-TAO paper,
>> Cassandra
>>> is likely able to support multiple distinct transaction layers that
>> operate
>>> independently. I would encourage you to file a CEP to explore how we can
>>> meet these distinct use cases, but I consider them to be niche. I expect
>>> that a majority of our user base desire strict serializable isolation,
>> and
>>> certainly no less than serializable isolation, to augment the existing
>>> weaker isolation offered by quorum reads and writes.
>>> 
>>> I would tangentially note that we are not an AP database under normal
>>> recommended operation. A minority in any network partition cannot reach
>>> QUORUM, so under recommended 

Re: [DISCUSS] CEP-15: General Purpose Transactions

[Blake Eggleston]

Hi Jake,

> 1.  Will this effort eventually replace consistency levels in C*?  I ask
> because one of the shortcomings of our paxos today is
> it can be easily mixed with non serialized consistencies and therefore
> users commonly break consistency by for example reading at CL.ONE while
> also
> using LWTs.

This will likely require CLs to be specified at the schema level for tables 
using multi partition transactions. I’d expect this to be available for other 
tables, but not required.

> 2. What structural changes are planned to support an external dependency
> project like this?  Are there some high level interfaces you expect the
> project to adhere to?

There will be some interfaces that need to be implemented in C* to support the 
library. You can find the current interfaces in the accord.api package, but 
these were written to support some initial testing, and not intended for 
integration into C* as is. Things are pretty fluid right now and will be 
rewritten / refactored multiple times over the next few months.

Thanks,

Blake


> On Sun, Sep 5, 2021 at 10:33 AM bened...@apache.org 
> wrote:
> 
>> Wiki:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-15%3A+General+Purpose+Transactions
>> Whitepaper:
>> https://cwiki.apache.org/confluence/download/attachments/188744725/Accord.pdf
>> <
>> https://cwiki.apache.org/confluence/download/attachments/188744725/Accord.pdf?version=1=1630847736966=v2
>>> 
>> Prototype: https://github.com/belliottsmith/accord
>> 
>> Hi everyone, I’d like to propose this CEP for adoption by the community.
>> 
>> Cassandra has benefitted from LWTs for many years, but application
>> developers that want to ensure consistency for complex operations must
>> either accept the scalability bottleneck of serializing all related state
>> through a single partition, or layer a complex state machine on top of the
>> database. These are sophisticated and costly activities that our users
>> should not be expected to undertake. Since distributed databases are
>> beginning to offer distributed transactions with fewer caveats, it is past
>> time for Cassandra to do so as well.
>> 
>> This CEP proposes the use of several novel techniques that build upon
>> research (that followed EPaxos) to deliver (non-interactive) general
>> purpose distributed transactions. The approach is outlined in the wikipage
>> and in more detail in the linked whitepaper. Importantly, by adopting this
>> approach we will be the _only_ distributed database to offer global,
>> scalable, strict serializable transactions in one wide area round-trip.
>> This would represent a significant improvement in the state of the art,
>> both in the academic literature and in commercial or open source offerings.
>> 
>> This work has been partially realised in a prototype. This partial
>> prototype has been verified against Jepsen.io’s Maelstrom library and
>> dedicated in-tree strict serializability verification tools, but much work
>> remains for the work to be production capable and integrated into Cassandra.
>> 
>> I propose including the prototype in the project as a new source
>> repository, to be developed as a standalone library for integration into
>> Cassandra. I hope the community sees the important value proposition of
>> this proposal, and will adopt the CEP after this discussion, so that the
>> library and its integration into Cassandra can be developed in parallel and
>> with the involvement of the wider community.
>> 
> 
> 
> -- 
> http://twitter.com/tjake


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0.1

[Blake Eggleston]

+1

> On Sep 1, 2021, at 4:54 AM, Sam Tunnicliffe  wrote:
> 
> Proposing the test build of Cassandra 4.0.1 for release.
> 
> sha1: 6709111ed007a54b3e42884853f89cabd38e4316
> Git: 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0.1-tentative
> Maven Artifacts: 
> https://repository.apache.org/content/repositories/orgapachecassandra-1247/org/apache/cassandra/cassandra-all/4.0.1/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and 
> repositories, are available here: 
> https://dist.apache.org/repos/dist/dev/cassandra/4.0.1/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has 
> tested the build is invited to vote. Votes by PMC members are considered 
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> [1]: CHANGES.txt: 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0.1-tentative
> [2]: NEWS.txt: 
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0.1-tentative
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-14: Paxos Improvements

[Blake Eggleston]

+1

> On Aug 27, 2021, at 12:48 PM, bened...@apache.org wrote:
> 
> Hi everyone, I’m proposing this CEP for approval.
> 
> Proposal: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-14%3A+Paxos+Improvements
> Discussion: 
> https://lists.apache.org/thread.html/r1af3da2d875ef93479e3874072ee651f406b96c915759c7968d3266e%40%3Cdev.cassandra.apache.org%3E
> 
> The vote will be open for 72 hours.
> Votes by committers are considered binding.
> A vote passes if there are at least three binding +1s and no binding vetoes.
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Repair Improvement Proposal

[Blake Eggleston]

+1 from me, any improvement in this area would be great.

It would be nice if this could include visibility into repair streams, but just 
exposing the repair state will be a big improvement.

> On Aug 25, 2021, at 5:46 PM, David Capwell  wrote:
> 
> Now that 4.0 is out, I want to bring up improving repair again (earlier
> thread
> http://mail-archives.apache.org/mod_mbox/cassandra-commits/201911.mbox/%3cjira.13266448.1572997299000.99567.1572997440...@atlassian.jira%3E),
> specifically the following two JIRAs:
> 
> 
> CASSANDRA-15566 - Repair coordinator can hang under some cases
> 
> CASSANDRA-15399 - Add ability to track state in repair
> 
> 
> Right now repair has an issue if any message is lost, which leads to hung
> or timed out repairs; in addition there is a large lack of visibility into
> what is going on, and can be even harder if you wish to join coordinator
> with participant state.
> 
> 
> I propose the following changes to improve our current repair subsystem:
> 
> 
> 
>   1. New tracking system for coordinator and participants (covered by
>   CASSANDRA-15399).  This system will expose progress on each instance and
>   expose this information for internal access as well as external users
>   2. Add retries to specific stages of coordination, such as prepare and
>   validate.  In order to do these retries we first need to know what the
>   state is for the participant which has yet to reply, this will leverage
>   CASSANDRA-15399 to see what's going on (has the prepare been seen?  Is
>   validation running? Did it complete?).  In addition to checking the
>   state, we will need to store the validation MerkleTree, this allows for
>   coordinator to fetch if goes missing (can be dropped in route to
>   coordinator or even on the coordinator).
> 
> 
> What is not in scope?
> 
>   - Rewriting all of Repair; the idea is specific "small" changes can fix
>   80% of the issues
>   - Handle coordinator node failure.  Being able to recover from a failed
>   coordinator should be possible after the above work is done, so is seen as
>   tangental and can be done later
>   - Recovery from a downed participant.  Similar to the previous bullet,
>   with the state being tracked this acts as a kind of checkpoint, so future
>   work can come in to handle recovery
>   - Handling "too large" range. Ideally we should add an ability to split
>   the coordination into sub repairs, but this is not the goal of this work.
>   - Overstreaming.  This is a byproduct of the previous "not in scope"
>   bullet, and/or large partitions; so is tangental to this work
> 
> 
> Wanted to share here before starting this work again; let me know if there
> are any concerns or feedback!


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] CEP-10: Cluster and Code Simulations

[Blake Eggleston]

+1

> On Jul 27, 2021, at 9:21 PM, Scott Andreas  wrote:
> 
> +1 nb
> 
> 
> From: Sam Tunnicliffe 
> Sent: Tuesday, July 27, 2021 12:54 AM
> To: dev@cassandra.apache.org
> Subject: Re: [VOTE] CEP-10: Cluster and Code Simulations
> 
> +1
> 
>> On 26 Jul 2021, at 11:51, bened...@apache.org wrote:
>> 
>> Proposing the CEP-10 (Cluster and Code Simulations) for adoption
>> 
>> Proposal: 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-10%3A+Cluster+and+Code+Simulations
>> Discussion: 
>> https://lists.apache.org/thread.html/rc908165994b15a29ef9c17b0b1205b2abc5bd38228b5a0117e442104%40%3Cdev.cassandra.apache.org%3E
>> 
>> The vote will be open for 72 hours.
>> Votes by PMC members are considered binding. A
>> vote passes if there are at least three binding +1s and no binding vetoes.
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0.0 (third time is the charm)

[Blake Eggleston]

+1

> On Jul 23, 2021, at 6:39 AM, Branimir Lambov  
> wrote:
> 
> +1
> 
>> On Fri, Jul 23, 2021 at 4:15 PM Aleksey Yeschenko 
>> wrote:
>> 
>> +1
>> 
 On 23 Jul 2021, at 14:03, Joshua McKenzie  wrote:
>>> 
>>> +1
>>> 
>>> On Fri, Jul 23, 2021 at 8:07 AM Dinesh Joshi >> 
>>> wrote:
>>> 
 +1
 
 
> On Jul 23, 2021, at 4:56 AM, Paulo Motta 
 wrote:
> 
> +1
> 
>> Em sex., 23 de jul. de 2021 às 08:37, Andrés de la Peña <
>> a.penya.gar...@gmail.com> escreveu:
>> 
>> +1
>> 
>>> On Fri, 23 Jul 2021 at 11:56, Sam Tunnicliffe 
>> wrote:
>>> 
>>> +1
>>> 
 On 22 Jul 2021, at 23:40, Brandon Williams <
 brandonwilli...@apache.org
>>> 
>>> wrote:
 
 I am proposing the test build of Cassandra 4.0.0 for release.
 
 sha1: 902b4d31772eaa84f05ffdc1e4f4b7a66d5b17e6
 Git:
>>> 
>> 
 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0.0-tentative
 Maven Artifacts:
 
>>> 
>> 
 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1244/org/apache/cassandra/cassandra-all/4.0.0/
 
 The Source and Build Artifacts, and Debian and RPM packages and
 repositories are available here:
 https://dist.apache.org/repos/dist/dev/cassandra/4.0.0/
 
 The vote will be open for 72 hours (longer if needed). Everyone who
 has tested the build is invited to vote. Votes by PMC members are
 considered binding. A vote passes if there are at least three
>> binding
 +1s and no -1's.
 
 [1]: CHANGES.txt:
 
>>> 
>> 
 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0.0-tentative
 [2]: NEWS.txt:
>>> 
>> 
 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0.0-tentative
 
 
>> -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 
> 
> -- 
> Branimir Lambov
> e. branimir.lam...@datastax.com
> w. www.datastax.com

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0.0 (take2)

[Blake Eggleston]

+1

> On Jul 14, 2021, at 8:21 AM, Aleksey Yeschenko  wrote:
> 
> +1
> 
>> On 14 Jul 2021, at 15:37, Jonathan Ellis  wrote:
>> 
>> +1
>> 
>>> On Tue, Jul 13, 2021 at 5:14 PM Mick Semb Wever  wrote:
>>> 
>>> Proposing the test build of Cassandra 4.0.0 for release.
>>> 
>>> sha1: 924bf92fab1820942137138c779004acaf834187
>>> Git:
>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0.0-tentative
>>> Maven Artifacts:
>>> 
>>> https://repository.apache.org/content/repositories/orgapachecassandra-1242/org/apache/cassandra/cassandra-all/4.0.0/
>>> 
>>> The Source and Build Artifacts, and the Debian and RPM packages and
>>> repositories, are available here:
>>> https://dist.apache.org/repos/dist/dev/cassandra/4.0.0/
>>> 
>>> The vote will be open for 72 hours (longer if needed). Everyone who
>>> has tested the build is invited to vote. Votes by PMC members are
>>> considered binding. A vote passes if there are at least three binding
>>> +1s and no -1's.
>>> 
>>> [1]: CHANGES.txt:
>>> 
>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0.0-tentative
>>> [2]: NEWS.txt:
>>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0.0-tentative
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
>> -- 
>> Jonathan Ellis
>> co-founder, http://www.datastax.com
>> @spyced
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-rc1 (take2)

[Blake Eggleston]

+1

> On Apr 21, 2021, at 2:25 PM, Scott Andreas  wrote:
> 
> +1nb, thank you!
> 
> 
> From: Ekaterina Dimitrova 
> Sent: Wednesday, April 21, 2021 12:23 PM
> To: dev@cassandra.apache.org
> Subject: Re: [VOTE] Release Apache Cassandra 4.0-rc1 (take2)
> 
> +1 and thanks everyone for all the hard work
> 
> Checked:
> - gpg signatures
> - sha checksums
> - binary convenience artifact runs
> - src convenience artifacts builds with one command, and runs
> - deb and rpm install and run
> 
>> On Wed, 21 Apr 2021 at 14:57, Michael Semb Wever  wrote:
>> 
>> 
>>> The vote will be open for 72 hours (longer if needed). Everyone who
>>> has tested the build is invited to vote. Votes by PMC members are
>>> considered binding. A vote passes if there are at least three binding
>>> +1s and no -1's.
>> 
>> 
>> +1
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-rc1

[Blake Eggleston]

+1

> On Mar 29, 2021, at 6:05 AM, Mick Semb Wever  wrote:
> 
> Proposing the test build of Cassandra 4.0-rc1 for release.
> 
> sha1: 2facbc97ea215faef1735d9a3d5697162f61bc8c
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-rc1-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1234/org/apache/cassandra/cassandra-all/4.0-rc1/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/4.0-rc1/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> Known issues with this release, that are planned to be fixed in 4.0-rc2, are
> - four files were missing copyright headers,
> - LICENSE and NOTICE contain additional unneeded information,
> - jar files under lib/ in the source artefact.
> 
> These issues are actively being worked on, along with our expectations that
> the ASF makes the policy around them more explicit so it is clear exactly
> what is required of us.
> 
> 
> [1]: CHANGES.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-rc1-tentative
> [2]: NEWS.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0-rc1-tentative


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release dtest-api 0.0.7

[Blake Eggleston]

+1, sorry for the html barf :)

> On Dec 3, 2020, at 9:53 AM, Blake Eggleston  
> wrote:
> 
> Proposing the test build of in-jvm dtest API 0.0.7 for 
> release.
> 
> Repository:https://gitbox.apache.org/repos/asf?p=cassandra-in-jvm-dtest-api.git;a=shortlog;h=refs/tags/0.0.7
> 
> Candidate 
> SHA:https://github.com/apache/cassandra-in-jvm-dtest-api/commit/d5174b1f44b7d9cb919d4975b4d437041273c09c
> tagged with 0.0.7
> Artifact:https://repository.apache.org/content/repositories/orgapachecassandra-1225/org/apache/cassandra/dtest-api/0.0.7/
> 
> Key signature: 9E66CEC6106D578D0B1EB9BFF1000962B7F6840C
> 
> Changes since last release:
> 
>   * CASSANDRA-16136: Add Metrics to instance API
>   * CASSANDRA-16272: Nodetool assert apis do not include the new
> stdout and stderr in the failure message
> 
> The vote will be open for 24 hours. Everyone who has tested the build
> is invited to vote. Votes by PMC members are considered binding. A
> vote passes if there are at least three binding +1s.
> 
> -- Alex
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release dtest-api 0.0.7

[Blake Eggleston]

Proposing the test build of in-jvm dtest API 0.0.7 for release.

Repository:https://gitbox.apache.org/repos/asf?p=cassandra-in-jvm-dtest-api.git;a=shortlog;h=refs/tags/0.0.7

Candidate 
SHA:https://github.com/apache/cassandra-in-jvm-dtest-api/commit/d5174b1f44b7d9cb919d4975b4d437041273c09c
tagged with 0.0.7
Artifact:https://repository.apache.org/content/repositories/orgapachecassandra-1225/org/apache/cassandra/dtest-api/0.0.7/

Key signature: 9E66CEC6106D578D0B1EB9BFF1000962B7F6840C

Changes since last release:

  * CASSANDRA-16136: Add Metrics to instance API
  * CASSANDRA-16272: Nodetool assert apis do not include the new
stdout and stderr in the failure message

The vote will be open for 24 hours. Everyone who has tested the build
is invited to vote. Votes by PMC members are considered binding. A
vote passes if there are at least three binding +1s.

-- Alex

Re: [DISCUSS] CASSANDRA-12126: LWTs correcteness and performance

[Blake Eggleston]

+1 to correctness, and I like the yaml idea

> On Nov 23, 2020, at 4:20 AM, Paulo Motta  wrote:
> 
> +1 to defaulting for correctness.
> 
> In addition to that, how about making it a mandatory cassandra.yaml
> property defaulting to correctness? This would make upgrades with an old
> cassandra.yaml fail unless an option is explicitly specified, making
> operators aware of the issue and forcing them to make a choice.
> 
>> Em seg., 23 de nov. de 2020 às 07:30, Benjamin Lerer <
>> benjamin.le...@datastax.com> escreveu:
>> 
>> Thank you very much to everybody that provided feedback. It helped a lot to
>> limit our options.
>> 
>> Unfortunately, it seems that some poor soul (me, really!!!) will have to
>> make the final call between #3 and #4.
>> 
>> If I reformulate the question to: Do we default to *correctness *or to
>> *performance*?
>> 
>> I would choose to default to *correctness*.
>> 
>> Of course the situation is more complex than that but it seems that
>> somebody has to make a call and live with it. It seems to me that being
>> blamed for choosing correctness is easier to live with ;-)
>> 
>> Benjamin
>> 
>> PS: I tried to push the choice on Sylvain but he dodged the bullet.
>> 
>> On Sat, Nov 21, 2020 at 12:30 AM Benedict Elliott Smith <
>> bened...@apache.org>
>> wrote:
>> 
>>> I think I meant #4 __‍♂️
>>> 
>>> On 20/11/2020, 21:11, "Blake Eggleston" 
>>> wrote:
>>> 
>>>I’d also prefer #3 over #4
>>> 
>>>> On Nov 20, 2020, at 10:03 AM, Benedict Elliott Smith <
>>> bened...@apache.org> wrote:
>>>> 
>>>> Well, I expressed a preference for #3 over #4, particularly for
>> the
>>> 3.x series.  However at this point, I think the lack of a clear project
>>> decision means we can punt it back to you and Sylvain to make the final
>>> call.
>>>> 
>>>> On 20/11/2020, 16:23, "Benjamin Lerer" <
>> benjamin.le...@datastax.com>
>>> wrote:
>>>> 
>>>>   I will try to summarize the discussion to clarify the outcome.
>>>> 
>>>>   Mick is in favor of #4
>>>>   Summanth is in favor of #4
>>>>   Sylvain answer was not clear for me. I understood it like I
>>> prefer #3 to #4
>>>>   and I am also fine with #1
>>>>   Jeff is in favor of #3 and will understand #4
>>>>   David is in favor #3 (fix bug and add flag to roll back to old
>>> behavior) in
>>>>   4.0 and #4 in 3.0 and 3.11
>>>> 
>>>>   Do not hesitate to correct me if I misunderstood your answer.
>>>> 
>>>>   Based on these answers it seems clear that most people prefer to
>>> go for #3
>>>>   or #4.
>>>> 
>>>>   The choice between #3 (fix correctness opt-in to current
>>> behavior) and #4
>>>>   (current behavior opt-in to correctness) is a bit less clear
>>> specially if
>>>>   we consider the 3.X branches or 4.0.
>>>> 
>>>>   Does anybody as some idea on how to choose between those 2
>>> choices or some
>>>>   extra opinions on #3 versus #4?
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>>>   On Wed, Nov 18, 2020 at 9:45 PM David Capwell <
>>> dcapw...@gmail.com> wrote:
>>>>> 
>>>>> I feel that #4 (fix bug and add flag to roll back to old behavior)
>>> is best.
>>>>> 
>>>>> About the alternative implementation, I am fine adding it to 3.x
>>> and 4.0,
>>>>> but should treat it as a different path disabled by default that
>>> you can
>>>>> opt-into, with a plan to opt-in by default "eventually".
>>>>> 
>>>>> On Wed, Nov 18, 2020 at 11:10 AM Benedict Elliott Smith <
>>>>> bened...@apache.org>
>>>>> wrote:
>>>>> 
>>>>>> Perhaps there might be broader appetite to weigh in on which
>> major
>>>>>> releases we might target for work that fixes the correctness bug
>>> without
>>>>>> serious performance regression?
>>>>>> 
>>>>>> i.e., if we were to fix the correctness bug now, introducing a
>>> serious
>>>>>> performance regression (either opt-in or opt-out), but were to
>>> land work
>>>>>>

Re: [DISCUSS] CASSANDRA-12126: LWTs correcteness and performance

[Blake Eggleston]

I’d also prefer #3 over #4

> On Nov 20, 2020, at 10:03 AM, Benedict Elliott Smith  
> wrote:
> 
> Well, I expressed a preference for #3 over #4, particularly for the 3.x 
> series.  However at this point, I think the lack of a clear project decision 
> means we can punt it back to you and Sylvain to make the final call.
> 
> On 20/11/2020, 16:23, "Benjamin Lerer"  wrote:
> 
>I will try to summarize the discussion to clarify the outcome.
> 
>Mick is in favor of #4
>Summanth is in favor of #4
>Sylvain answer was not clear for me. I understood it like I prefer #3 to #4
>and I am also fine with #1
>Jeff is in favor of #3 and will understand #4
>David is in favor #3 (fix bug and add flag to roll back to old behavior) in
>4.0 and #4 in 3.0 and 3.11
> 
>Do not hesitate to correct me if I misunderstood your answer.
> 
>Based on these answers it seems clear that most people prefer to go for #3
>or #4.
> 
>The choice between #3 (fix correctness opt-in to current behavior) and #4
>(current behavior opt-in to correctness) is a bit less clear specially if
>we consider the 3.X branches or 4.0.
> 
>Does anybody as some idea on how to choose between those 2 choices or some
>extra opinions on #3 versus #4?
> 
> 
> 
> 
> 
> 
>>On Wed, Nov 18, 2020 at 9:45 PM David Capwell  wrote:
>> 
>> I feel that #4 (fix bug and add flag to roll back to old behavior) is best.
>> 
>> About the alternative implementation, I am fine adding it to 3.x and 4.0,
>> but should treat it as a different path disabled by default that you can
>> opt-into, with a plan to opt-in by default "eventually".
>> 
>> On Wed, Nov 18, 2020 at 11:10 AM Benedict Elliott Smith <
>> bened...@apache.org>
>> wrote:
>> 
>>> Perhaps there might be broader appetite to weigh in on which major
>>> releases we might target for work that fixes the correctness bug without
>>> serious performance regression?
>>> 
>>> i.e., if we were to fix the correctness bug now, introducing a serious
>>> performance regression (either opt-in or opt-out), but were to land work
>>> without this problem for 5.0, would there be appetite to backport this
>> work
>>> to any of 4.0, 3.11 or 3.0?
>>> 
>>> 
>>> On 18/11/2020, 18:31, "Jeff Jirsa"  wrote:
>>> 
>>>This is complicated and relatively few people on earth understand it,
>>> so
>>>having little feedback is mostly expected, unfortunately.
>>> 
>>>My normal emotional response is "correctness is required, opt-in to
>>>performance improvements that sacrifice strict correctness", but I'm
>>> also
>>>sure this is going to surprise people, and would understand / accept
>> #4
>>>(default to current, opt-in to correct).
>>> 
>>> 
>>>On Wed, Nov 18, 2020 at 4:54 AM Benedict Elliott Smith <
>>> bened...@apache.org>
>>>wrote:
>>> 
 It doesn't seem like there's much enthusiasm for any of the options
 available here...
 
 On 12/11/2020, 14:37, "Benedict Elliott Smith" <
>> bened...@apache.org
 
 wrote:
 
> Is the new implementation a separate, distinctly modularized
>>> new
 body of work
 
It’s primarily a distinct, modularised and new body of work,
>>> however
 there is some shared code that has been modified - namely
>>> PaxosState, in
 which legacy code is maintained but modified for compatibility, and
>>> the
 system.paxos table (which receives a new column, and slightly
>>> modified
 serialization code).  It is conceptually an optimised version of
>> the
 existing algorithm.
 
If there's a chance of being of value to 4.0, I can try to put
>>> up a
 patch next week alongside a high level description of the changes.
 
> But a performance regression is a regression, I'm not
>>> shrugging it
 off.
 
I don't want to give the impression I'm shrugging off the
>>> correctness
 issue either. It's a serious issue to fix, but since all successful
>>> updates
 to the database are linearizable, I think it's likely that many
 applications behave correctly with the present semantics, or at
>> least
 encounter only transient errors. No doubt many also do not, but I
>>> have no
 idea of the ratio.
 
The regression isn't itself a simple issue either - depending
>> on
>>> the
 topology and message latencies it is not difficult to produce
>>> inescapable
 contention, i.e. guaranteed timeouts - that might persist as long
>> as
 clients continue to retry. It could be quite a serious degradation
>> of
 service to impose on our users.
 
I don't pretend to know the correct way to make a decision
>>> balancing
 these considerations, but I am perhaps more concerned about
>> imposing
 service outages than I am temporarily maintaining semantics our
>>> users have
 apparently accepted for years - though I absolutely share your
 embarrassment there.
 
 
On 12/11/2020, 12:41, 

Re: [VOTE] Accept the Harry donation

[Blake Eggleston]

+1

> On Sep 16, 2020, at 2:45 AM, Mick Semb Wever  wrote:
> 
> This vote is about officially accepting the Harry donation from Alex Petrov
> and Benedict Elliott Smith, that was worked on in CASSANDRA-15348.
> 
> The Incubator IP Clearance has been filled out at
> http://incubator.apache.org/ip-clearance/apache-cassandra-harry.html
> 
> This vote is a required part of the IP Clearance process. It follows the
> same voting rules as releases, i.e. from the PMC a minimum of three +1s and
> no -1s.
> 
> Please cast your votes:
>   [ ] +1 Accept the contribution into Cassandra
>   [ ] -1 Do not


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] Change style guide to recommend use of @Override

[Blake Eggleston]

+1

> On Sep 1, 2020, at 11:27 AM, David Capwell  wrote:
> 
> Currently our style guide recommends to avoid using @Override and updates
> intellij's code style to exclude it by default; I would like to propose we
> change this recommendation to use it and to update intellij's style to
> include it by default.
> 
> @Override is used by javac to enforce that a method is in fact overriding
> from an abstract class or an interface and if this stops being true (such
> as a refactor happens) then a compiler error is thrown; when we default to
> excluding, it makes it harder to detect that a refactor catches all
> implementations and can lead to subtle and hard to track down bugs.
> 
> This proposal is for new code and would not be to go rewrite all code at
> once, but would recommend new code adopt this style, and to pull old code
> forward which is related to changes being made (similar to our stance on
> imports).
> 
> If people are ok with this, I will file a JIRA, update the docs, and
> update intellij's formatting.
> 
> Thanks for your time!


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-beta2

[Blake Eggleston]

+1

> On Aug 28, 2020, at 7:18 AM, Mick Semb Wever  wrote:
> 
> Proposing the test build of Cassandra 4.0-beta2 for release.
> 
> sha1: 56eadf2004399a80f0733041cacf03839832249a
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-beta2-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1218/org/apache/cassandra/cassandra-all/4.0-beta2/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/4.0-beta2/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> [1]: CHANGES.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-beta2-tentative
> [2]: NEWS.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0-beta2-tentative


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 3.0.22

[Blake Eggleston]

+1

> On Aug 28, 2020, at 6:09 AM, Mick Semb Wever  wrote:
> 
> Proposing the test build of Cassandra 3.0.22 for release.
> 
> sha1: 45331bb612dc7847efece7e26cdd0b376bd11249
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.22-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1216/org/apache/cassandra/cassandra-all/3.0.22/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/3.0.22/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> [1]: CHANGES.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.22-tentative
> [2]: NEWS.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/3.0.22-tentative


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 2.1.22

[Blake Eggleston]

+1

> On Aug 28, 2020, at 8:55 AM, Jeff Jirsa  wrote:
> 
> +1
> 
> 
> On Fri, Aug 28, 2020 at 8:42 AM Mick Semb Wever  wrote:
> 
>> Proposing the test build of Cassandra 2.1.22 for release.
>> 
>> sha1: 94e9149c22f6a7772c0015e1b1ef2e2961155c0a
>> Git:
>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.1.22-tentative
>> Maven Artifacts:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1214
>> /org/apache/cassandra/cassandra-all/2.1.22/
>> 
>> 
>> The Source and Build Artifacts, and the Debian and RPM packages and
>> repositories, are available here:
>> https://dist.apache.org/repos/dist/dev/cassandra/2.1.22/
>> 
>> The vote will be open for 72 hours (longer if needed). Everyone who has
>> tested the build is invited to vote. Votes by PMC members are considered
>> binding. A vote passes if there are at least three binding +1s and no -1's.
>> 
>> [1]: CHANGES.txt:
>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/2.1.22-tentative
>> [2]: NEWS.txt:
>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/2.1.22-tentative
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 2.2.18

[Blake Eggleston]

+1

> On Aug 28, 2020, at 5:44 AM, Mick Semb Wever  wrote:
> 
> Proposing the test build of Cassandra 2.2.18 for release.
> 
> sha1: d4938cf4e488a9ef3ac48164a3e946f16255d721
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.2.18-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1215/org/apache/cassandra/cassandra-all/2.2.18/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/2.2.18/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> [1]: CHANGES.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.18-tentative
> [2]: NEWS.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/2.2.18-tentative


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 3.11.8

[Blake Eggleston]

+1

> On Aug 28, 2020, at 6:37 AM, Mick Semb Wever  wrote:
> 
> Proposing the test build of Cassandra 3.11.8 for release.
> 
> sha1: 8b29b698630960a0ebb2c695cc5b21dee4686d09
> Git:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.8-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1217/org/apache/cassandra/cassandra-all/3.11.8/
> 
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/3.11.8/
> 
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
> 
> [1]: CHANGES.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.8-tentative
> [2]: NEWS.txt:
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/3.11.8-tentative


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Check in on CASSANDRA-15393

[Blake Eggleston]

Caleb is currently working through his second round of review, and Marcus said 
he's about halfway through his review as of this morning. So I'd expect it to 
be committed within a week or so.

> On Aug 27, 2020, at 5:09 PM, Joshua McKenzie  wrote:
> 
> Is there an ETA on them landing? The later, the more risk to stability of
> GA due to lack of time soaking.
> 
> On Thu, Aug 27, 2020 at 4:01 PM Blake Eggleston
>  wrote:
> 
>> Hi dev@,
>> 
>> Mick asked that I check in w/ the dev list about CASSANDRA-15393. There's
>> some concern regarding the patch and it's suitability for inclusion in
>> 4.0-beta.
>> 
>> CASSANDRA-15393 reduces garbage created by compaction and the read paths
>> by about 25%. It's part of CASSANDRA-15387, which, including this patch,
>> reduces garbage from the read and compaction paths by about 50%.
>> CASSANDRA-15393 does this by supporting byte array backed cell and
>> clustering types, which is acheived by abstracting the backing type
>> (ByteBuffer/byte[]) from the serialization logic.
>> 
>> To avoid paying the allocation cost of adding a container object,
>> singleton "accessor" objects are used to operate on the actual data. See
>> here for an example:
>> https://gist.github.com/bdeggleston/52910225b817a8d54353125ca03f521d
>> 
>> Mick and Robert Stupp have raised a few concerns, summarized below:
>> 
>> 1. The patch is large (208 files / ~3.5k LOC)
>> 2. Concerns about impact on stability
>> 3. Parameterizing cell/clustering value types in this way makes
>> ClassCastExceptions possible.
>> 4. implications of feature freeze
>> 
>> The patch is large, but the vast majority of it is adding type parameters
>> to things. The changes here are wide, but not deep. The most complex parts
>> are the collection serializers and other places where we're now having to
>> do offset bookkeeping. These should be carefully reviewed, but they
>> shouldn't be too difficult to verify and I've added some randomized tests
>> to check them against a wide range of schemas. I'll also run some diff
>> tests against clusters internally.
>> 
>> Parameterizing cell and clustering values does make ClassCastExceptions
>> possible, but java's type system guards against this for the most part.
>> Regarding the feature freeze, I don't think it applies to performance
>> improvements.
>> 
>> Back to the point about stability though: in pracice, compaction gc is a
>> major contributor to cluster instability. In my experience, about 30% of
>> availability issues are gc related. Also, compaction gc tends to be the
>> limiting factor for repair, host replacements, and other topology changes,
>> which limits how quickly you can recover from other issues. So the patch
>> does add some risk, but I think it's a net win for stability.
>> 
>> Thoughts?
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Check in on CASSANDRA-15393

[Blake Eggleston]

Hi dev@,

Mick asked that I check in w/ the dev list about CASSANDRA-15393. There's some 
concern regarding the patch and it's suitability for inclusion in 4.0-beta.

CASSANDRA-15393 reduces garbage created by compaction and the read paths by 
about 25%. It's part of CASSANDRA-15387, which, including this patch, reduces 
garbage from the read and compaction paths by about 50%. CASSANDRA-15393 does 
this by supporting byte array backed cell and clustering types, which is 
acheived by abstracting the backing type (ByteBuffer/byte[]) from the 
serialization logic. 

To avoid paying the allocation cost of adding a container object, singleton 
"accessor" objects are used to operate on the actual data. See here for an 
example: https://gist.github.com/bdeggleston/52910225b817a8d54353125ca03f521d

Mick and Robert Stupp have raised a few concerns, summarized below:

1. The patch is large (208 files / ~3.5k LOC)
2. Concerns about impact on stability
3. Parameterizing cell/clustering value types in this way makes 
ClassCastExceptions possible.
4. implications of feature freeze

The patch is large, but the vast majority of it is adding type parameters to 
things. The changes here are wide, but not deep. The most complex parts are the 
collection serializers and other places where we're now having to do offset 
bookkeeping. These should be carefully reviewed, but they shouldn't be too 
difficult to verify and I've added some randomized tests to check them against 
a wide range of schemas. I'll also run some diff tests against clusters 
internally.

Parameterizing cell and clustering values does make ClassCastExceptions 
possible, but java's type system guards against this for the most part. 
Regarding the feature freeze, I don't think it applies to performance 
improvements.

Back to the point about stability though: in pracice, compaction gc is a major 
contributor to cluster instability. In my experience, about 30% of availability 
issues are gc related. Also, compaction gc tends to be the limiting factor for 
repair, host replacements, and other topology changes, which limits how quickly 
you can recover from other issues. So the patch does add some risk, but I think 
it's a net win for stability.

Thoughts?
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-beta1 (take2)

[Blake Eggleston]

+1

> On Jul 20, 2020, at 9:56 AM, Jon Haddad  wrote:
> 
> +1, thanks Mick for rerolling.
> 
> On Mon, Jul 20, 2020 at 6:42 AM Joshua McKenzie 
> wrote:
> 
>> +1
>> 
>> On Mon, Jul 20, 2020 at 8:51 AM Jake Luciani  wrote:
>> 
>>> +1
>>> 
>>> On Mon, Jul 20, 2020 at 8:08 AM Andrés de la Peña <
>>> a.penya.gar...@gmail.com>
>>> wrote:
>>> 
 +1 (nb)
 
 On Mon, 20 Jul 2020 at 12:58, João Reis 
>> wrote:
 
> +1 (nb)
> 
> The drivers smoke test suite looks good:
> 
> 
> 
 
>>> 
>> https://ci.appveyor.com/project/DataStax/cassandra-drivers-smoke-test/builds/34194004
> 
> Mick Semb Wever  escreveu no dia sábado, 18/07/2020
>>> à(s)
> 00:27:
> 
>> Proposing the test build of Cassandra 4.0-beta1 for release.
>> 
>> sha1: 972da6fcffa87b3a1684362a2bab97db853372d8
>> Git:
>> 
>> 
> 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-beta1-tentative
>> Maven Artifacts:
>> 
>> 
> 
 
>>> 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1211/org/apache/cassandra/cassandra-all/4.0-beta1/
>> 
>> The Source and Build Artifacts, and the Debian and RPM packages and
>> repositories, are available here:
>> https://dist.apache.org/repos/dist/dev/cassandra/4.0-beta1/
>> 
>> The vote will be open for 60 hours (longer if needed). I've taken
>> 12
> hours
>> off the normal 72 hours and this follows closely after the initial
>> 4.0-beta1 vote. Everyone who has tested the build is invited to
>> vote.
> Votes
>> by PMC members are considered binding. A vote passes if there are
>> at
> least
>> three binding +1s and no -1s.
>> 
>> Eventual publishing and announcement of the 4.0-beta1 release will
>> be
>> coordinated, as described in
>> 
>> 
> 
 
>>> 
>> https://lists.apache.org/thread.html/r537fe799e7d5e6d72ac791fdbe9098ef0344c55400c7f68ff65abe51%40%3Cdev.cassandra.apache.org%3E
>> 
>> [1]: CHANGES.txt:
>> 
>> 
> 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-beta1-tentative
>> [2]: NEWS.txt:
>> 
>> 
> 
 
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0-beta1-tentative
>> 
> 
 
>>> 
>>> 
>>> --
>>> http://twitter.com/tjake
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Media coordination (was: [VOTE] Release Apache Cassandra 4.0-beta1)

[Blake Eggleston]

Characterizing alternate or conflicting points of view as assuming bad 
intentions without justification is both unproductive and unhealthy for the 
project.

> On Jul 20, 2020, at 9:14 AM, Joshua McKenzie  wrote:
> 
> This kind of back and forth isn't productive for the project so I'm not
> taking this discussion further. Just want to call it out here so you or
> others aren't left waiting for a reply.
> 
> We can agree to disagree.
> 
> On Mon, Jul 20, 2020 at 11:59 AM Benedict Elliott Smith 
> wrote:
> 
>> Firstly, that is a very strong claim that in this particular case is
>> disputed by the facts.  You made a very specific claim that the delay was
>> "risking our currently lined up coordination with journalists and other
>> channels". I am not the only person to interpret this as implying
>> coordination with journalists, contingent on a release schedule not agreed
>> by the PMC.  This was based on semantics only; as far as I can tell, no
>> intentions or assumptions have entered into this debate, except on your
>> part.
>> 
>>> Which is the definition of not assuming positive intent.
>> 
>> Secondly, this is not the definition of positive intent.  Positive intent
>> only indicates that you "mean well"
>> 
>> Thirdly, in many recent disputes about governance, you have made a
>> negative claim about my behaviour, or ascribed negative connotations to
>> statements I have made; this is a very thinly veiled example, as I am
>> clearly the object of this criticism.  I think it has reached a point where
>> I can perhaps legitimately claim that you are not assuming positive intent?
>> 
>>> motives, incentives ... little to do with reality
>> 
>> It feels like we should return to this earlier discussion, since you
>> appear to feel it is incomplete?  At the very least you seem to have taken
>> the wrong message from my statements, and it is perhaps negatively
>> colouring our present interactions.
>> 
>> 
>> On 20/07/2020, 15:59, "Joshua McKenzie"  wrote:
>> 
>>> 
>>> If you are criticised, it is often because of the action you took;
>> 
>>Actually, in this case and many others it's because of people's
>> unfounded
>>assumptions about motives, incentives, and actions taken and has
>> little to
>>do with reality. Which is the definition of not assuming positive
>> intent.
>> 
>>On Mon, Jul 20, 2020 at 10:41 AM Benedict Elliott Smith <
>> bened...@apache.org>
>>wrote:
>> 
>>> Thanks Sally, really appreciate your insight.
>>> 
>>> To respond to the community discourse around this:
>>> 
 Keep your announcement plans ... private: limit discussions to the
>> PMC
>>> 
>>> This is all that I was asking and expecting: if somebody is making
>>> commitments on behalf of the community (such as that a release can be
>>> expected on day X), this should be coordinated with the PMC.  While
>> it
>>> seems to transpire that no such commitments were made, had they been
>> made
>>> without the knowledge of the PMC this would in my view be
>> problematic.
>>> This is not at all like development work, as has been alleged, since
>> that
>>> only takes effect after public agreement by the community.
>>> 
>>> IMO, in general, public engagements should be run past the PMC as a
>> final
>>> pre-flight check regardless of any commitment being made, as the PMC
>> should
>>> have visibility into these activities and have the opportunity to
>> influence
>>> them.
>>> 
 There has been nothing about this internally at DS
>>> 
>>> I would ask that you refrain from making such claims, unless you can
>> be
>>> certain that you would have been privy to all such internal
>> discussions.
>>> 
 there's really no reason not to assume best intentions here
>>> 
>>> This is a recurring taking point, that I wish we would retire except
>> where
>>> a clear assumption of bad faith has been made.  If you are
>> criticised, it
>>> is often because of the action you took; any intention you had may be
>>> irrelevant to the criticism.  In this case, when you act on behalf
>> of the
>>> community, your intentions are insufficient: you must have the
>> community's
>>> authority to act.
>>> 
>>> 
>>> On 20/07/2020, 14:00, "Sally Khudairi"  wrote:
>>> 
>>>Hello everyone --Mick pinged me about this; I wanted to respond
>>> on-list for efficacy.
>>> 
>>>We've had dozens of companies successfully help Apache Projects
>> and
>>> their communities help spread the word on their projects with their
>> PR and
>>> marketing teams. Here are some best practices:
>>> 
>>>1) Timing. Ensure that the Project has announced the project
>> milestone
>>> first to their lists as well as announce@ before any media coverage
>> takes
>>> place. If you're planning to time the announcements to take place in
>>> tandem, be careful with embargoes, as not everyone is able to honor
>> them.
>>> We've been burned in the past with this.
>>> 
>>>2) Messaging. Keep your announcement plans and draft press
>> releases,
>>> etc., private: limit 

Re: Media coordination (was: [VOTE] Release Apache Cassandra 4.0-beta1)

[Blake Eggleston]

I don't think Benedict mentioned anything about people's motives or intentions, 
he simply had a concern about how marketing timelines became a factor in a 
release vote without the approval of the PMC. I think this is a reasonable 
concern, and doesn't mean that he's assuming bad intentions. That's my reading 
at least, although maybe I missed something?

> On Jul 20, 2020, at 7:58 AM, Joshua McKenzie  wrote:
> 
>> 
>> If you are criticised, it is often because of the action you took;
> 
> Actually, in this case and many others it's because of people's unfounded
> assumptions about motives, incentives, and actions taken and has little to
> do with reality. Which is the definition of not assuming positive intent.
> 
> On Mon, Jul 20, 2020 at 10:41 AM Benedict Elliott Smith 
> wrote:
> 
>> Thanks Sally, really appreciate your insight.
>> 
>> To respond to the community discourse around this:
>> 
>>> Keep your announcement plans ... private: limit discussions to the PMC
>> 
>> This is all that I was asking and expecting: if somebody is making
>> commitments on behalf of the community (such as that a release can be
>> expected on day X), this should be coordinated with the PMC.  While it
>> seems to transpire that no such commitments were made, had they been made
>> without the knowledge of the PMC this would in my view be problematic.
>> This is not at all like development work, as has been alleged, since that
>> only takes effect after public agreement by the community.
>> 
>> IMO, in general, public engagements should be run past the PMC as a final
>> pre-flight check regardless of any commitment being made, as the PMC should
>> have visibility into these activities and have the opportunity to influence
>> them.
>> 
>>> There has been nothing about this internally at DS
>> 
>> I would ask that you refrain from making such claims, unless you can be
>> certain that you would have been privy to all such internal discussions.
>> 
>>> there's really no reason not to assume best intentions here
>> 
>> This is a recurring taking point, that I wish we would retire except where
>> a clear assumption of bad faith has been made.  If you are criticised, it
>> is often because of the action you took; any intention you had may be
>> irrelevant to the criticism.  In this case, when you act on behalf of the
>> community, your intentions are insufficient: you must have the community's
>> authority to act.
>> 
>> 
>> On 20/07/2020, 14:00, "Sally Khudairi"  wrote:
>> 
>>Hello everyone --Mick pinged me about this; I wanted to respond
>> on-list for efficacy.
>> 
>>We've had dozens of companies successfully help Apache Projects and
>> their communities help spread the word on their projects with their PR and
>> marketing teams. Here are some best practices:
>> 
>>1) Timing. Ensure that the Project has announced the project milestone
>> first to their lists as well as announce@ before any media coverage takes
>> place. If you're planning to time the announcements to take place in
>> tandem, be careful with embargoes, as not everyone is able to honor them.
>> We've been burned in the past with this.
>> 
>>2) Messaging. Keep your announcement plans and draft press releases,
>> etc., private: limit discussions to the PMC. Drafting announcements on
>> public lists, such as user@, whilst inclusive, may inadvertently expose
>> your news prematurely to the press, bloggers, and others before its ready.
>> This can be detrimental to having your news scooped before you actually
>> announce it, or conversely, having the news come out and nobody is
>> interested in covering it as it's been leaking for a while. We've also been
>> burned in the past with this. Synching messaging is also helpful to ensure
>> that the PMC speaks with a unified voice: the worst thing that can happen
>> is having someone say one thing in the media and another member of the PMC
>> saying something else, even if it's their personal opinion. Fragmentation
>> helps no-one. This recently happened with a Project on a rather
>> controversial topic, so the press was excited to see dissent within the
>> community as it gave them more to report about. Keep things co
>> ol: don't be the feature cover of a gossip tabloid.
>> 
>>3) Positioning. It's critical that whomever is speaking on behalf of
>> the Project identify themselves as such. This means that the PMC needs to
>> have a few spokespeople lined up in case of any media queries, and that the
>> spokespeople supporting the project are from different organizations so you
>> can . I cannot stress enough the need to exhibit diversity, even if
>> everyone working on the media/marketing side is from a single organization
>> --the ASF comes down hard on companies that "own" projects: we take
>> vendor-neutrality very seriously. What's worked well with organizations
>> that have pitched the press on behalf of a project is to pitch the project
>> news, have spokespeople from other organizations speak 

Re: [DISCUSS] Future of MVs

[Blake Eggleston]

+1 for deprecation and removal (assuming a credible plan to fix them doesn't 
materialize)

> On Jun 30, 2020, at 12:43 PM, Jon Haddad  wrote:
> 
> A couple days ago when writing a separate email I came across this DataStax
> blog post discussing MVs [1].  Imagine my surprise when I noticed the date
> was five years ago...
> 
> While at TLP, I helped numerous customers move off of MVs, mostly because
> they affected stability of clusters in a horrific way.  The most telling
> project involved helping someone create new tables to manage 1GB of data
> because the views performed so poorly they made the cluster unresponsive
> and unusable.  Despite being around for five years, they've seen very
> little improvement that makes them usable for non trivial, non laptop
> workloads.
> 
> Since the original commits, it doesn't look like there's been much work to
> improve them, and they're yet another feature I ended up saying "just don't
> use".  I haven't heard any plans to improve them in any meaningful way -
> either to address their issues with performance or the inability to repair
> them.
> 
> The original contributor of MVs (Carl Yeksigian) seems to have disappeared
> from the project, meaning we have a broken feature without a maintainer,
> and no plans to fix it.
> 
> As we move forward with the 4.0 release, we should consider this an
> opportunity to deprecate materialized views, and remove them in 5.0.  We
> should take this opportunity to learn from the mistake and raise the bar
> for new features to undergo a much more thorough run the wringer before
> merging.
> 
> I'm curious what folks think - am I way off base here?  Am I missing a JIRA
> that can magically fix the issues with performance, availability &
> correctness?
> 
> [1]
> https://www.datastax.com/blog/2015/06/new-cassandra-30-materialized-views
> [2] https://issues.apache.org/jira/browse/CASSANDRA-6477


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Project governance wiki doc (take 2)

[Blake Eggleston]

+1

> On Jun 20, 2020, at 8:12 AM, Joshua McKenzie  wrote:
> 
> Link to doc:
> https://cwiki.apache.org/confluence/display/CASSANDRA/Apache+Cassandra+Project+Governance
> 
> Change since previous cancelled vote:
> "A simple majority of this electorate becomes the low-watermark for votes
> in favour necessary to pass a motion, with new PMC members added to the
> calculation."
> 
> This previously read "super majority". We have lowered the low water mark
> to "simple majority" to balance strong consensus against risk of stall due
> to low participation.
> 
> 
>   - Vote will run through 6/24/20
>   - pmc votes considered binding
>   - simple majority of binding participants passes the vote
>   - committer and community votes considered advisory
> 
> Lastly, I propose we take the count of pmc votes in this thread as our
> initial roll call count for electorate numbers and low watermark
> calculation on subsequent votes.
> 
> Thanks again everyone (and specifically Benedict and Jon) for the time and
> collaboration on this.
> 
> ~Josh


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-alpha4

[Blake Eggleston]

+1

> On Apr 14, 2020, at 5:09 AM, e.dimitr...@gmail.com wrote:
> 
> I also can’t see them. I think it matters to which interface is the link. 
> 
> And +1 from me, thanks! 
> 
>> On 14 Apr 2020, at 7:53, Erick Ramirez  wrote:
>> 
>> 
>>> 
>>> 
 All java8 UTs, jvmdtests and dtests pass
 
>>> https://circleci.com/workflow-run/d7b3f62d-c9ad-43d6-9152-2655e27feccb?signup-404=true
>>> 
>>> 
>>> Is anyone else able to see this^ circleci page?
>>> For me, it never loads, and isn't the first time I've been unable to
>>> see others' circleci results.
>>> 
>> 
>> All that shows up for me is:
>> 
>> Workflows  >>  null  >>  null  >> null
>> 0 jobs in this workflow
>> 
>> 
>> and a spinning widget.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-alpha2

[Blake Eggleston]

+1

> On Oct 25, 2019, at 8:57 AM, Jeff Jirsa  wrote:
> 
> +1
> 
> 
> On Fri, Oct 25, 2019 at 8:06 AM Jon Haddad  wrote:
> 
>> +1
>> 
>> On Fri, Oct 25, 2019 at 10:18 AM Sam Tunnicliffe  wrote:
>> 
>>> +1
>>> 
 On 24 Oct 2019, at 18:26, Michael Shuler 
>> wrote:
 
 I propose the following artifacts for release as 4.0-alpha2.
 
 sha1: ca928a49c68186bdcd57dea8b10c30991c6a3c55
 Git:
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-alpha2-tentative
 Artifacts:
>>> 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1185/org/apache/cassandra/apache-cassandra/4.0-alpha2/
 Staging repository:
>>> 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1185/
 
 The Debian and RPM packages are available here:
>>> http://people.apache.org/~mshuler
 
 The vote will be open for 72 hours (longer if needed).
 
 [1]: CHANGES.txt:
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-alpha2-tentative
 [2]: NEWS.txt:
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0-alpha2-tentative
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 3.11.5

[Blake Eggleston]

+1

> On Oct 25, 2019, at 8:58 AM, Jeff Jirsa  wrote:
> 
> +1
> 
> 
> On Fri, Oct 25, 2019 at 7:18 AM Sam Tunnicliffe  wrote:
> 
>> +1
>> 
>>> On 24 Oct 2019, at 18:26, Michael Shuler  wrote:
>>> 
>>> I propose the following artifacts for release as 3.11.5.
>>> 
>>> sha1: b697af87f8e1b20d22948390d516dba1fbb9eee7
>>> Git:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.11.5-tentative
>>> Artifacts:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1184/org/apache/cassandra/apache-cassandra/3.11.5/
>>> Staging repository:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1184/
>>> 
>>> The Debian and RPM packages are available here:
>> http://people.apache.org/~mshuler
>>> 
>>> The vote will be open for 72 hours (longer if needed).
>>> 
>>> [1]: CHANGES.txt:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.5-tentative
>>> [2]: NEWS.txt:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/3.11.5-tentative
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 2.2.15

[Blake Eggleston]

+1

> On Oct 25, 2019, at 8:57 AM, Jeff Jirsa  wrote:
> 
> +1
> 
> 
> On Fri, Oct 25, 2019 at 7:18 AM Sam Tunnicliffe  wrote:
> 
>> +1
>> 
>>> On 24 Oct 2019, at 18:25, Michael Shuler  wrote:
>>> 
>>> I propose the following artifacts for release as 2.2.15.
>>> 
>>> sha1: 4ee4ceea28a1cb77b283c7ce0135340ddff02086
>>> Git:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/2.2.15-tentative
>>> Artifacts:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1179/org/apache/cassandra/apache-cassandra/2.2.15/
>>> Staging repository:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1179/
>>> 
>>> The Debian and RPM packages are available here:
>> http://people.apache.org/~mshuler
>>> 
>>> The vote will be open for 72 hours (longer if needed).
>>> 
>>> [1]: CHANGES.txt:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.15-tentative
>>> [2]: NEWS.txt:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/2.2.15-tentative
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 3.0.19

[Blake Eggleston]

+1

> On Oct 25, 2019, at 8:58 AM, Jeff Jirsa  wrote:
> 
> +1
> 
> 
> On Fri, Oct 25, 2019 at 7:18 AM Sam Tunnicliffe  wrote:
> 
>> +1
>> 
>>> On 24 Oct 2019, at 18:25, Michael Shuler  wrote:
>>> 
>>> I propose the following artifacts for release as 3.0.19.
>>> 
>>> sha1: a81bfd6b7db3a373430b3c4e8f4e930b199796f0
>>> Git:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/3.0.19-tentative
>>> Artifacts:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1183/org/apache/cassandra/apache-cassandra/3.0.19/
>>> Staging repository:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1183/
>>> 
>>> The Debian and RPM packages are available here:
>> http://people.apache.org/~mshuler
>>> 
>>> The vote will be open for 72 hours (longer if needed).
>>> 
>>> [1]: CHANGES.txt:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.19-tentative
>>> [2]: NEWS.txt:
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/3.0.19-tentative
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Can we kick off a release?

[Blake Eggleston]

Looks like 15193 has been committed. Are we waiting on anything else before 
cutting the next set of releases?

> On Oct 8, 2019, at 1:11 PM, Jon Haddad  wrote:
> 
> I forgot to mention, we should also release alpha2 of 4.0.
> 
> 
> On Tue, Oct 8, 2019 at 1:04 PM Michael Shuler 
> wrote:
> 
>> Thanks Sam, I'm following #15193 and should catch the status change there.
>> 
>> Michael
>> 
>> On Tue, Oct 8, 2019 at 6:17 AM Sam Tunnicliffe  wrote:
>>> 
>>> CASSANDRA-15193 just got +1’d yesterday and would be good to include in
>> the 3.0 and 3.11 releases. If you don’t mind holding off while I add a
>> cqlsh test and merge it, that’d be good.
>>> 
>>> Thanks,
>>> Sam
>>> 
 On 7 Oct 2019, at 22:54, Michael Shuler 
>> wrote:
 
 Will do! I probably won't get this done this evening, so will send out
 the emails tomorrow.
 
 Thanks,
 Michael
 
 On Mon, Oct 7, 2019 at 2:37 PM Jon Haddad  wrote:
> 
> Michael,
> 
> Would you mind kicking off builds and starting a vote thread for the
>> latest
> 2.2, 3.0 and 3.11 builds?
> 
> Much appreciated,
> Jon
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: fixing paging state for 4.0

[Blake Eggleston]

Sorry, I misread your earlier email. Yes, there are drivers that do mixed 
protocol versions. Not sure if the 4.0 java driver does, but at least one 
previous version did.

> On Sep 24, 2019, at 7:19 PM, Blake Eggleston  
> wrote:
> 
> Yes, but if a client is connected to 2 different nodes, and is using a 
> different protocol for each, the paging state formats aren’t going to match 
> if it tries to use the paging date from one connection on the other.
> 
>> On Sep 24, 2019, at 7:14 PM, J. D. Jordan  wrote:
>> 
>> It is inherently versioned by the protocol version being used for the 
>> connection.
>> 
>>> On Sep 24, 2019, at 9:06 PM, Jon Haddad  wrote:
>>> 
>>> The problem is that the payload isn't versioned, because the individual
>>> fields aren't really part of the protocol.  I think the long term fix
>>> should be to add the fields of the paging state to the protocol itself
>>> rather than have it just be some serialized blob.  Then we don't have to
>>> deal with separately versioning the paging state.
>>> 
>>> I think recognizing max int as special number that just means "a lot" is
>>> fine for now till we have time to rework it is a reasonable approach.
>>> 
>>> Jon
>>> 
>>>> On Tue, Sep 24, 2019 at 6:52 PM J. D. Jordan 
>>>> wrote:
>>>> 
>>>> Are their drivers that try to do mixed protocol version connections?  If
>>>> so that would be a mistake on the drivers part if it sent the new paging
>>>> state to an old server.  Pretty easily protected against in said driver
>>>> when it implements support for the new protocol version.  The payload is
>>>> opaque, but that doesn’t mean a driver would send the new payload to an old
>>>> server.
>>>> 
>>>> Many of the drivers I have looked at don’t do mixed version connections.
>>>> If they start at a higher version they will not connect to older nodes that
>>>> don’t support it. Or they will connect to the newer nodes with the older
>>>> protocol version. In either of those cases there is no problem.
>>>> 
>>>> Protocol changes aside, I would suggest fixing the bug starting back on
>>>> 3.x by changing the meaning of MAX. Whether or not the limit is switched to
>>>> a var int in a bumped protocol version.
>>>> 
>>>> -Jeremiah
>>>> 
>>>> 
>>>>> On Sep 24, 2019, at 8:28 PM, Blake Eggleston
>>>>  wrote:
>>>>> 
>>>>> Right, that's the problem with changing the paging state format. It
>>>> doesn't work in mixed mode.
>>>>> 
>>>>>> On Sep 24, 2019, at 4:47 PM, Jeremiah Jordan 
>>>> wrote:
>>>>>> 
>>>>>> Clients do negotiate the protocol version they use when connecting. If
>>>> the server bumped the protocol version then this larger paging state could
>>>> be part of the new protocol version. But that doesn’t solve the problem for
>>>> existing versions.
>>>>>> 
>>>>>> The special treatment of Integer.MAX_VALUE can be done back to 3.x and
>>>> fix the bug in all versions, letting users requests to receive all of their
>>>> data.  Which realistically is probably what someone who sets the protocol
>>>> level query limit to Integer.MAX_VALUE is trying to do.
>>>>>> 
>>>>>> -Jeremiah
>>>>>> 
>>>>>>>> On Sep 24, 2019, at 4:09 PM, Blake Eggleston
>>>>  wrote:
>>>>>>> 
>>>>>>> Right, mixed version clusters. The opaque blob isn't versioned, and
>>>> there isn't an opportunity for min version negotiation that you have with
>>>> the messaging service. The result is situations where a client begins a
>>>> read on one node, and attempts to read the next page from a different node
>>>> over a protocol version where the paging state serialization format has
>>>> changed. This causes an exception deserializing the paging state and the
>>>> read fails.
>>>>>>> 
>>>>>>> There are ways around this, but they're not comprehensive (I think),
>>>> and they're much more involved than just interpreting Integer.MAX_VALUE as
>>>> unlimited. The "right" solution would be for the paging state to be
>>>> deserialized/serialized on the client side, but that won't happen in 4.0.
>>>>>

Re: fixing paging state for 4.0

[Blake Eggleston]

Yes, but if a client is connected to 2 different nodes, and is using a 
different protocol for each, the paging state formats aren’t going to match if 
it tries to use the paging date from one connection on the other.

> On Sep 24, 2019, at 7:14 PM, J. D. Jordan  wrote:
> 
> It is inherently versioned by the protocol version being used for the 
> connection.
> 
>> On Sep 24, 2019, at 9:06 PM, Jon Haddad  wrote:
>> 
>> The problem is that the payload isn't versioned, because the individual
>> fields aren't really part of the protocol.  I think the long term fix
>> should be to add the fields of the paging state to the protocol itself
>> rather than have it just be some serialized blob.  Then we don't have to
>> deal with separately versioning the paging state.
>> 
>> I think recognizing max int as special number that just means "a lot" is
>> fine for now till we have time to rework it is a reasonable approach.
>> 
>> Jon
>> 
>>> On Tue, Sep 24, 2019 at 6:52 PM J. D. Jordan 
>>> wrote:
>>> 
>>> Are their drivers that try to do mixed protocol version connections?  If
>>> so that would be a mistake on the drivers part if it sent the new paging
>>> state to an old server.  Pretty easily protected against in said driver
>>> when it implements support for the new protocol version.  The payload is
>>> opaque, but that doesn’t mean a driver would send the new payload to an old
>>> server.
>>> 
>>> Many of the drivers I have looked at don’t do mixed version connections.
>>> If they start at a higher version they will not connect to older nodes that
>>> don’t support it. Or they will connect to the newer nodes with the older
>>> protocol version. In either of those cases there is no problem.
>>> 
>>> Protocol changes aside, I would suggest fixing the bug starting back on
>>> 3.x by changing the meaning of MAX. Whether or not the limit is switched to
>>> a var int in a bumped protocol version.
>>> 
>>> -Jeremiah
>>> 
>>> 
>>>> On Sep 24, 2019, at 8:28 PM, Blake Eggleston
>>>  wrote:
>>>> 
>>>> Right, that's the problem with changing the paging state format. It
>>> doesn't work in mixed mode.
>>>> 
>>>>> On Sep 24, 2019, at 4:47 PM, Jeremiah Jordan 
>>> wrote:
>>>>> 
>>>>> Clients do negotiate the protocol version they use when connecting. If
>>> the server bumped the protocol version then this larger paging state could
>>> be part of the new protocol version. But that doesn’t solve the problem for
>>> existing versions.
>>>>> 
>>>>> The special treatment of Integer.MAX_VALUE can be done back to 3.x and
>>> fix the bug in all versions, letting users requests to receive all of their
>>> data.  Which realistically is probably what someone who sets the protocol
>>> level query limit to Integer.MAX_VALUE is trying to do.
>>>>> 
>>>>> -Jeremiah
>>>>> 
>>>>>>> On Sep 24, 2019, at 4:09 PM, Blake Eggleston
>>>  wrote:
>>>>>> 
>>>>>> Right, mixed version clusters. The opaque blob isn't versioned, and
>>> there isn't an opportunity for min version negotiation that you have with
>>> the messaging service. The result is situations where a client begins a
>>> read on one node, and attempts to read the next page from a different node
>>> over a protocol version where the paging state serialization format has
>>> changed. This causes an exception deserializing the paging state and the
>>> read fails.
>>>>>> 
>>>>>> There are ways around this, but they're not comprehensive (I think),
>>> and they're much more involved than just interpreting Integer.MAX_VALUE as
>>> unlimited. The "right" solution would be for the paging state to be
>>> deserialized/serialized on the client side, but that won't happen in 4.0.
>>>>>> 
>>>>>>>> On Sep 24, 2019, at 1:12 PM, Jon Haddad  wrote:
>>>>>>>> 
>>>>>>>> What's the pain point?  Is it because of mixed version clusters or is
>>> there
>>>>>>> something else that makes it a problem?
>>>>>>> 
>>>>>>>> On Tue, Sep 24, 2019 at 11:03 AM Blake Eggleston
>>>>>>>>  wrote:
>>>>>>>> 
>>>

Re: fixing paging state for 4.0

[Blake Eggleston]

Right, that's the problem with changing the paging state format. It doesn't 
work in mixed mode.

> On Sep 24, 2019, at 4:47 PM, Jeremiah Jordan  wrote:
> 
> Clients do negotiate the protocol version they use when connecting. If the 
> server bumped the protocol version then this larger paging state could be 
> part of the new protocol version. But that doesn’t solve the problem for 
> existing versions.
> 
> The special treatment of Integer.MAX_VALUE can be done back to 3.x and fix 
> the bug in all versions, letting users requests to receive all of their data. 
>  Which realistically is probably what someone who sets the protocol level 
> query limit to Integer.MAX_VALUE is trying to do.
> 
> -Jeremiah
> 
>> On Sep 24, 2019, at 4:09 PM, Blake Eggleston  
>> wrote:
>> 
>> Right, mixed version clusters. The opaque blob isn't versioned, and there 
>> isn't an opportunity for min version negotiation that you have with the 
>> messaging service. The result is situations where a client begins a read on 
>> one node, and attempts to read the next page from a different node over a 
>> protocol version where the paging state serialization format has changed. 
>> This causes an exception deserializing the paging state and the read fails.
>> 
>> There are ways around this, but they're not comprehensive (I think), and 
>> they're much more involved than just interpreting Integer.MAX_VALUE as 
>> unlimited. The "right" solution would be for the paging state to be 
>> deserialized/serialized on the client side, but that won't happen in 4.0.
>> 
>>> On Sep 24, 2019, at 1:12 PM, Jon Haddad  wrote:
>>> 
>>> What's the pain point?  Is it because of mixed version clusters or is there
>>> something else that makes it a problem?
>>> 
>>>> On Tue, Sep 24, 2019 at 11:03 AM Blake Eggleston
>>>>  wrote:
>>>> 
>>>> Changing paging state format is kind of a pain since the driver treats it
>>>> as an opaque blob. I'd prefer we went with Sylvain's suggestion to just
>>>> interpret Integer.MAX_VALUE as "no limit", which would be a lot simpler to
>>>> implement.
>>>> 
>>>>> On Sep 24, 2019, at 10:44 AM, Jon Haddad  wrote:
>>>>> 
>>>>> I'm working with a team who just ran into CASSANDRA-14683 [1], which I
>>>>> didn't realize was an issue till now.
>>>>> 
>>>>> Anyone have an interest in fixing full table pagination?  I'm not sure of
>>>>> the full implications of changing the int to a long in the paging stage.
>>>>> 
>>>>> https://urldefense.proofpoint.com/v2/url?u=https-3A__issues.apache.org_jira_browse_CASSANDRA-2D14683=DwIFAg=adz96Xi0w1RHqtPMowiL2g=CNZK3RiJDLqhsZDG6FQGnXn8WyPRCQhp4x_uBICNC0g=6_gWDV_kv-TQJ8GyBlYfcrhPGl7WmGYGEJ9ET6rPARo=LcYkbQwf4gzl8tnMcVbFKr3PeZ_u8mHHnXTBRWtIZFU=
>>>>>  
>>>> 
>>>> 
>>>> -
>>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>>> 
>>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: fixing paging state for 4.0

[Blake Eggleston]

Right, mixed version clusters. The opaque blob isn't versioned, and there isn't 
an opportunity for min version negotiation that you have with the messaging 
service. The result is situations where a client begins a read on one node, and 
attempts to read the next page from a different node over a protocol version 
where the paging state serialization format has changed. This causes an 
exception deserializing the paging state and the read fails.

There are ways around this, but they're not comprehensive (I think), and 
they're much more involved than just interpreting Integer.MAX_VALUE as 
unlimited. The "right" solution would be for the paging state to be 
deserialized/serialized on the client side, but that won't happen in 4.0.

> On Sep 24, 2019, at 1:12 PM, Jon Haddad  wrote:
> 
> What's the pain point?  Is it because of mixed version clusters or is there
> something else that makes it a problem?
> 
> On Tue, Sep 24, 2019 at 11:03 AM Blake Eggleston
>  wrote:
> 
>> Changing paging state format is kind of a pain since the driver treats it
>> as an opaque blob. I'd prefer we went with Sylvain's suggestion to just
>> interpret Integer.MAX_VALUE as "no limit", which would be a lot simpler to
>> implement.
>> 
>>> On Sep 24, 2019, at 10:44 AM, Jon Haddad  wrote:
>>> 
>>> I'm working with a team who just ran into CASSANDRA-14683 [1], which I
>>> didn't realize was an issue till now.
>>> 
>>> Anyone have an interest in fixing full table pagination?  I'm not sure of
>>> the full implications of changing the int to a long in the paging stage.
>>> 
>>> https://issues.apache.org/jira/browse/CASSANDRA-14683
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Attention to serious bug CASSANDRA-15081

[Blake Eggleston]

This looks like a dupe of CASSANDRA-15086, which has been committed and will be 
included in 3.0.19.

> On Sep 11, 2019, at 5:10 PM, Cameron Zemek  wrote:
> 
> Have had multiple customer hit this CASSANDRA-15081 issue now, where
> upgrading from older versions the sstables contain an unknown column (its
> not present in the dropped_columns in the schema)
> 
> This bug is serious as reads return incorrect results and if you run scrub
> it will drop the row. So hoping to bring it some attention to have the
> issue resolved. Note I have included a patch that I think does not cause
> any regressions elsewhere.
> 
> Regards,
> Cameron


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: fixing paging state for 4.0

[Blake Eggleston]

Changing paging state format is kind of a pain since the driver treats it as an 
opaque blob. I'd prefer we went with Sylvain's suggestion to just interpret 
Integer.MAX_VALUE as "no limit", which would be a lot simpler to implement.

> On Sep 24, 2019, at 10:44 AM, Jon Haddad  wrote:
> 
> I'm working with a team who just ran into CASSANDRA-14683 [1], which I
> didn't realize was an issue till now.
> 
> Anyone have an interest in fixing full table pagination?  I'm not sure of
> the full implications of changing the int to a long in the paging stage.
> 
> https://issues.apache.org/jira/browse/CASSANDRA-14683


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Time for a new 3.0/3.11 release?

[Blake Eggleston]

Hi dev@,

Any objections to doing a new 3.0 and 3.11 release? Both branches have 
accumulated a decent number of changes since their last release, the highlights 
being improved merkle tree footprint, a gossip race, and a handful of 2.1 -> 
3.x upgrade bugs.

Thanks,

Blake


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Stabilising Internode Messaging in 4.0

[Blake Eggleston]

It seems like one of the main points of contention isn’t so much the the 
content of the patch, but more about the amount of review this patch has/will 
receive relative to its perceived risk. If it’s the latter, then I think it 
would be more effective to explain why that’s the case, and what level of 
review would be more appropriate.

I’m personally +0  on requiring additional review. I feel like the 3 people 
involved so far have sufficient expertise, and trust them to be responsible, 
including soliciting additional reviews if they feel they’re needed.

If dev@ does collectively want more eyes on this, I’d suggest we solicit 
reviews from people who are very familiar with the messaging code, and let them 
decide what additional work and documentation they’d need to make a review 
manageable, if any. Everyone has their own review style, and there’s no need to 
ask for a bunch of additional work if it’s not needed.

> On Apr 12, 2019, at 12:46 PM, Jordan West  wrote:
> 
> Since their seems to be an assumption that I haven’t read the code let me
> clarify: I am working on making time to be a reviewer on this and I have
> already spent a few hours with the patch before I sent any replies, likely
> more than most who are replying here. Again, because I disagree on
> non-technical matters does not mean I haven’t considered the technical. I
> am sharing what I think is necessary for the authors
> to make review higher quality. I will not compromise my review standards on
> a patch like this as I have said already. Telling me to review it to talk
> more about it directly ignores my feedback and requires me to acquiesce all
> of my concerns, which as I said I won’t do as a reviewer.
> 
> And yes I am arguing for changing how the Cassandra community approaches
> large patches. In the same way the freeze changed how we approached major
> releases and the decision to do so has been a net benefit as measured by
> quality and stability. Existing community members have already chimed in in
> support of things like better commit hygiene.
> 
> The past approaches haven’t prioritized quality and stability and it really
> shows. What I and others here are suggesting has worked all over our
> industry and is adopted by companies big (like google as i linked
> previously) and small (like many startups I and others have worked for).
> Everything we want to do: better testing, better review, better code, is
> made easier with better design review, better discussion, and more
> digestible patches among many of the other things suggested in this thread.
> 
> Jordan
> 
> On Fri, Apr 12, 2019 at 12:01 PM Benedict Elliott Smith 
> wrote:
> 
>> I would once again exhort everyone making these kinds of comment to
>> actually read the code, and to comment on Jira.  Preferably with a
>> justification by reference to the code for how or why it would improve the
>> patch.
>> 
>> As far as a design document is concerned, it’s very unclear what is being
>> requested.  We already had plans, as Jordan knows, to produce a wiki page
>> for posterity, and a blog post closer to release.  However, I have never
>> heard of this as a requirement for review, or for commit.  We have so far
>> taken two members of the community through the patch over video chat, and
>> would be more than happy to do the same for others.  So far nobody has had
>> any difficulty getting to grips with its structure.
>> 
>> If the project wants to modify its normal process for putting a patch
>> together, this is a whole different can of worms, and I am strongly -1.
>> I’m not sure what precedent we’re trying to set imposing arbitrary
>> constraints pre-commit for work that has already met the project’s
>> inclusion criteria.
>> 
>> 
>>> On 12 Apr 2019, at 18:58, Pavel Yaskevich  wrote:
>>> 
>>> I haven't actually looked at the code
>> 
>> 
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Stabilising Internode Messaging in 4.0

[Blake Eggleston]

Well said Josh. You’ve pretty much summarized my thoughts on this as well.

+1 to moving forward with this

> On Apr 11, 2019, at 10:15 PM, Joshua McKenzie  wrote:
> 
> As one of the two people that re-wrote all our unit tests to try and help
> Sylvain get 8099 out the door, I think it's inaccurate to compare the scope
> and potential stability impact of this work to the truly sweeping work that
> went into 8099 (not to downplay the scope and extent of this work here).
> 
> TBH, one of the big reasons we tend to drop such large PRs is the fact that
>> Cassandra's code is highly intertwined and it makes it hard to precisely
>> change things. We need to iterate towards interfaces that allow us to
>> iterate quickly and reduce the amount of highly intertwined code. It helps
>> with testing as well. I want us to have a meaningful discussion around it
>> before we drop a big PR.
> 
> This has been a huge issue with our codebase since at least back when I
> first encountered it five years ago. To date, while we have made progress
> on this front, it's been nowhere near sufficient to mitigate the issues in
> the codebase and allow for large, meaningful changes in smaller incremental
> patches or commits. Having yet another discussion around this (there have
> been many, many of them over the years) as a blocker for significant work
> to go into the codebase is an unnecessary and dangerous blocker. Not to say
> we shouldn't formalize a path to try and make incremental progress to
> improve the situation, far from it, but blocking other progress on a
> decade's worth of accumulated hygiene problems isn't going to make the
> community focus on fixing those problems imo, it'll just turn away
> contributions.
> 
> So let me second jd (and many others') opinion here: "it makes sense to get
> it right the first time, rather than applying bandaids to 4.0 and rewriting
> things for 4.next". And fwiw, asking people who have already done a huge
> body of work to reformat that work into a series of commits or to break up
> that work in a fashion that's more to the liking of people not involved in
> either the writing of the patch or reviewing of it doesn't make much sense
> to me. As I am neither an assignee nor reviewer on this contribution, I
> leave it up to the parties involved to do things professionally and with a
> high standard of quality. Admittedly, a large code change merging in like
> this has implications for rebasing on anyone else's work that's in flight,
> but be it one commit merged or 50, or be it one JIRA ticket or ten, the
> end-result is the same; any large contribution in any format will ripple
> outwards and require re-work from others in the community.
> 
> The one thing I *would* strongly argue for is performance benchmarking of
> the new messaging code on a representative sample of different
> general-purpose queries, LWT's, etc, preferably in a 3 node RF=3 cluster,
> plus a healthy suite of jmh micro-benches (assuming they're not already in
> the diff. If they are, disregard / sorry). From speaking with Aleksey
> offline about this work, my understanding is that that's something they
> plan on doing before putting a bow on things.
> 
> In the balance between "fear change because it destabilizes" and "go forth
> blindly into that dark night, rewriting All The Things", I think the
> Cassandra project's willingness to jettison the old and introduce the new
> has served it well in keeping relevant as the years have gone by. I'd hate
> to see that culture of progress get mired in a dogmatic adherence to
> requirements on commit counts, lines of code allowed / expected on a given
> patch set, or any other metrics that might stymie the professional needs of
> some of the heaviest contributors to the project.
> 
> On Wed, Apr 10, 2019 at 5:03 PM Oleksandr Petrov 
> wrote:
> 
>> Sorry to pick only a few points to address, but I think these are ones
>> where I can contribute productively to the discussion.
>> 
>>> In principle, I agree with the technical improvements you
>> mention (backpressure / checksumming / etc). These things should be there.
>> Are they a hard requirement for 4.0?
>> 
>> One thing that comes to mind is protocol versioning and consistency. If
>> changes adding checksumming and handshake do not make it to 4.0, we grow
>> the upgrade matrix and have to put changes to the separate protocol
>> version. I'm not sure how many other internode protocol changes we have
>> planned for 4.next, but this is definitely something we should keep in
>> mind.
>> 
>>> 2. We should not be measuring complexity in LoC with the exception that
>> all 20k lines *do need to be review* (not just the important parts and
>> because code refactoring tools have bugs too) and more lines take more
>> time.
>> 
>> Everything should definitely be reviewed. But with different rigour: one
>> thing is to review byte arithmetic and protocol formats and completely
>> different thing is to verify that Verb moved from one 

Re: Both Java 8 and Java 11 required for producing a tarball

[Blake Eggleston]

You may want to wait until CASSANDRA-14607 is finished before starting on 
14712. I think it will end up unwinding most of the stuff requiring building 
with dual jdks (either as part of that ticket or an immediate follow on).

I'm still working on making sure I haven't broken anything, but I'm currently 
able to build with a single jdk (8 or 11) without any problems. I also haven't 
run into any problems running a strictly jdk 8 build in java 11. I have more 
testing to do, but it seems ok so far.

> On Mar 13, 2019, at 4:14 PM, Stefan Miklosovic 
>  wrote:
> 
> Hi,
> 
> how do I assign myself to
> https://issues.apache.org/jira/browse/CASSANDRA-14712 ?
> 
> I read the doco here (1) but I think that workflow does not apply to
> cassandra-builds repo.
> 
> Should I do this first and then notify people? Until then it might happen
> that my time would be wasted as somebody else would start to work on that
> simultaneously.
> 
> (1) https://cassandra.apache.org/doc/latest/development/patches.html#patches
> 
> On Thu, 14 Mar 2019 at 08:46, Jordan West  wrote:
> 
>> A couple related JIRAs for reference:
>> 
>> https://issues.apache.org/jira/browse/CASSANDRA-14714
>> https://issues.apache.org/jira/browse/CASSANDRA-14712
>> 
>> On Wed, Mar 6, 2019 at 7:34 PM Michael Shuler 
>> wrote:
>> 
>>> On 3/6/19 7:10 PM, Stefan Miklosovic wrote:
 I am trying to build 4.0 from sources and prior to this I was doing
 
 ant artifacts
 
 in order to get distribution tarball to play with.
 
 If I understand this right, if I do not run Ant with Java 11,
 java.version.8 will be true so it will skip building tarballs.
>>> 
>>> Correct. You'll get a JDK8-only jar, but no full tar artifact set.
>>> 
 1) Why would one couldnt create a tarball while running on Java 8 only?
>>> 
>>> The build system needs a dual-JDK install to build the artifacts with
>>> support for each/both.
>>> 
 2) What is the current status of Java 11 / Java 8? Is it there just "to
>>> try
 it out if it runs with that" or are there different reasons behind it?
>>> 
>>> JDK8 runtime is the default, JDK11 runtime is optional, but supported.
>>> Here's the JIRA with all the details:
>>> https://issues.apache.org/jira/browse/CASSANDRA-9608
>>> 
>>> I just pushed a WIP branch to do a dual-JDK build via docker, since we
>>> need to work on this, too. (lines may wrap:)
>>> 
>>> git clone -b tar-artifact-build
>>> https://gitbox.apache.org/repos/asf/cassandra-builds.git
>>> 
>>> cd cassandra-builds/
>>> 
>>> docker build -t cass-build-tars -f docker/buster-image.docker docker/
>>> 
>>> docker run --rm -v `pwd`/dist:/dist `docker images -f
>>> label=org.cassandra.buildenv=buster -q` /home/build/build-tars.sh trunk
>>> 
>>> After all that, here's my tar artifacts:
>>> 
>>> (tar-artifact-build)mshuler@hana:~/git/cassandra-builds$ ls -l dist/
>>> total 94328
>>> -rw-r--r-- 1 mshuler mshuler 50385890 Mar  6 21:16
>>> apache-cassandra-4.0-SNAPSHOT-bin.tar.gz
>>> -rw-r--r-- 1 mshuler mshuler 46198947 Mar  6 21:16
>>> apache-cassandra-4.0-SNAPSHOT-src.tar.gz
>>> 
>>> Or you could drop a dual-JDK install on your machine, export the env
>>> vars you found and `ant artifacts` should produce the tars :)
>>> 
>>> --
>>> Kind regards,
>>> Michael
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
> 
> Stefan Miklosovic


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Modeling Time Series data

[Blake Eggleston]

This is a question for the user list.

> On Jan 11, 2019, at 1:51 PM, Akash Gangil  wrote:
> 
> Hi,
> 
> I have a data model where the partition key for a lot of tables is based on
> time
> (year, month, day, hour)
> 
> Would this create a hotspot in my cluster, given all the writes/reads would
> go to the same node for a given hour? Or does the cassandra storage engine
> also takes into account the table info like table name, when distributing
> the data?
> 
> If the above model would be a problem, what's the suggested way to solve
> this? Add tablename to partition key?
> 
> 
> 
> -- 
> Akash


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Change Jira Workflow

[Blake Eggleston]

+1

> On Dec 17, 2018, at 9:31 AM, jay.zhu...@yahoo.com.INVALID wrote:
> 
> +1
> 
>On Monday, December 17, 2018, 9:10:55 AM PST, Jason Brown 
>  wrote:  
> 
> +1.
> 
> On Mon, Dec 17, 2018 at 7:36 AM Michael Shuler 
> wrote:
> 
>> +1
>> 
>> --
>> Michael
>> 
>> On 12/17/18 9:19 AM, Benedict Elliott Smith wrote:
>>> I propose these changes <
>> https://cwiki.apache.org/confluence/display/CASSANDRA/JIRA+Workflow+Proposals>*
>> to the Jira Workflow for the project.  The vote will be open for 72 hours**.
>>> 
>>> I am, of course, +1.
>>> 
>>> * With the addendum of the mailing list discussion <
>> https://lists.apache.org/thread.html/e4668093169aa4ef52f2bea779333f04a0afde8640c9a79a8c86ee74@%3Cdev.cassandra.apache.org%3E>;
>> in case of any conflict arising from a mistake on my part in the wiki, the
>> consensus reached by polling the mailing list will take precedence.
>>> ** I won’t be around to close the vote, as I will be on vacation.
>> Everyone is welcome to ignore the result until I get back in a couple of
>> weeks, or if anybody is eager feel free to close the vote and take some
>> steps towards implementation.
>>> 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: JIRA Workflow Proposals

[Blake Eggleston]

1: A
2: +1
3: +1
4: +1
5: +1
6: +1

> On Dec 4, 2018, at 11:19 AM, Benedict Elliott Smith  
> wrote:
> 
> Sorry, 4. Is inconsistent.  First instance should be.
> 
>> - 4. Priorities: Keep ‘High' priority
> 
> 
>> On 4 Dec 2018, at 19:12, Benedict Elliott Smith > > wrote:
>> 
>> Ok, so after an initial flurry everyone has lost interest :)
>> 
>> I think we should take a quick poll (not a vote), on people’s positions on 
>> the questions raised so far.  If people could try to take the time to stake 
>> a +1/-1, or A/B, for each item, that would be really great.  This poll will 
>> not be the end of discussions, but will (hopefully) at least draw a line 
>> under the current open questions.
>> 
>> I will start with some verbiage, then summarise with options for everyone to 
>> respond to.  You can scroll to the summary immediately if you like.
>> 
>> - 1. Component: Multi-select or Cascading-select (i.e. only one component 
>> possible per ticket, but neater UX)
>> - 2. Labels: rather than litigate people’s positions, I propose we do the 
>> least controversial thing, which is to simply leave labels intact, and only 
>> supplement them with the new schema information.  We can later revisit if we 
>> decide it’s getting messy.
>> - 3. "First review completed; second review ongoing": I don’t think we need 
>> to complicate the process; if there are two reviews in flight, the first 
>> reviewer can simply comment that they are done when ready, and the second 
>> reviewer can move the status once they are done.  If the first reviewer 
>> wants substantive changes, they can move the status to "Change Request” 
>> before the other reviewer completes, if they like.  Everyone involved can 
>> probably negotiate this fairly well, but we can introduce some specific 
>> guidance on how to conduct yourself here in a follow-up.  
>> - 4. Priorities: Keep ‘High' priority
>> - 5. Mandatory Platform and Feature. Make mandatory by introducing new “All” 
>> and “None” (respectively) options, so always possible to select an option.
>> - 6. Environment field: Remove?
>> 
>> I think this captures everything that has been brought up so far, except for 
>> the suggestion to make "Since Version” a “Version” - but that needs more 
>> discussion, as I don’t think there’s a clear alternative proposal yet.
>> 
>> Summary:
>> 
>> 1: Component. (A) Multi-select; (B) Cascading-select
>> 2: Labels: leave alone +1/-1
>> 3: No workflow changes for first/second review: +1/-1
>> 4: Priorities: Including High +1/-1
>> 5: Mandatory Platform and Feature: +1/-1
>> 6: Remove Environment field: +1/-1
>> 
>> I will begin.
>> 
>> 1: A
>> 2: +1
>> 3: +1
>> 4: +1
>> 5: Don’t mind
>> 6: +1
>> 
>> 
>> 
>> 
>>> On 29 Nov 2018, at 22:04, Scott Andreas >>  >> >> wrote:
>>> 
>>> If I read Josh’s reply right, I think the suggestion is to periodically 
>>> review active labels and promote those that are demonstrably useful to 
>>> components (cf. folksonomy -> 
>>> taxonomy>>  
>>> >> >>). I 
>>> hadn’t read the reply as indicating that labels should be zero’d out 
>>> periodically. In any case, I agree that reviewing active labels and 
>>> re-evaluating our taxonomy from time to time sounds great; I don’t think 
>>> I’d zero them, though.
>>> 
>>> Responding to a few comments:
>>> 
>>> –––
>>> 
>>> – To Joey’s question about issues languishing in Triage: I like the idea of 
>>> an SLO for the “triage” state. I am happy to commit time and resources to 
>>> triaging newly-reported issues, and to JIRA pruning/gardening in general. I 
>>> spent part of the weekend before last adding components to a few hundred 
>>> open issues and preparing the Confluence reports mentioned in the other 
>>> thread. It was calming. We can also figure out how to rotate / share this 
>>> responsibility.
>>> 
>>> – Labels discussion: If we adopt a more structured component hierarchy to 
>>> treat as our primary method of organization, keep labels around for people 
>>> to use as they’d like (e.g., for custom JQL queries useful to their 
>>> workflows), and periodically promote those that are widely useful, I think 
>>> that sounds like a fine outcome.
>>> 
>>> – On Sankalp’s question of issue reporter / new contributor burden: I 
>>> actually think the pruning of fields on the “new issue form” makes 
>>> reporting issues easier and ensures that information we need is captured. 
>>> Having the triage step will also provide a nice task queue for screening 
>>> bugs, and ensures a contributor’s taken a look + screened appropriately 
>>> (rather than support requests immediately being marked “Critical/Blocker” 

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

Reading through the history Sankalp posted (I think it was originally posted by 
Joey?), I think part of the problem we’re having here is that we’re trying to 
solve at least 3 problems with a single solution. Also, I don’t think everyone 
has the same goals in mind. The issues we’re trying to solve are:


Repair scheduling - original proposal was for an in-process distributed 
scheduler to make cassandra eventually consistent without relying on external 
tools.

Sidecar - proposed as a helper co-process to make stuff like cluster wide 
nodetool command execution, health check, etc easier. I don’t think the 
original proposal mentioned repair.

Ops center like management application with a ui seems to have made it’s way 
into the mix at some point

These are all intended to make cassandra easier to operate, but they’re really 
separate features. It would be more productive to focus on each one as it’s own 
feature instead of trying to design a one size fits all and does everything 
management tool.

On September 12, 2018 at 6:25:11 PM, sankalp kohli (kohlisank...@gmail.com) 
wrote:

Here is a list of open discussion points from the voting thread. I think  
some are already answered but I will still gather these questions here.  

From several people:  
1. Vote is rushed and we need more time for discussion.  

From Sylvain  
2. About the voting process...I think that was addressed by Jeff Jirsa and  
deserves a separate thread as it is not directly related to this thread.  
3. Does the project need a side car.  

From Jonathan Haddad  
4. Are people doing +1 willing to contribute  

From Jonathan Ellis  
5. List of feature set, maturity, maintainer availability from Reaper or  
any other project being donated.  

Mick Semb Wever  
6. We should not vote on these things and instead build consensus.  

Open Questions from this thread  
7. What technical debts we are talking about in Reaper. Can someone give  
concrete examples.  
8. What is the timeline of donating Reaper to Apache Cassandra.  

On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli   
wrote:  

> (Using this thread and not the vote thread intentionally)  
> For folks talking about vote being rushed. I would use the email from  
> Joseph to show this is not rushed. There was no email on this thread for 4  
> months until I pinged.  
>  
>  
> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper to  
> come up with design goals for a repair scheduler that could work at Netflix  
> scale.  
>  
> ~Feb 2017: Netflix believes that the fundamental design gaps prevented us  
> from using Reaper as it relies heavily on remote JMX connections and  
> central coordination.  
>  
> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available  
> and distributed repair scheduling sidecar/tool. He is encouraged by  
> multiple committers to build repair scheduling into the daemon itself and  
> not as a sidecar so the database is truly eventually consistent.  
>  
> ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback at  
> NGCC, Vinay and myself prototype the distributed repair scheduler within  
> Priam and roll it out at Netflix scale.  
>  
> Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page  
> design document for adding repair scheduling to the daemon itself and open  
> the design up for feedback from the community. We get feedback from Alex,  
> Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals  
> to contribute Reaper at this point. We hear the consensus that the  
> community would prefer repair scheduling in a separate distributed sidecar  
> rather than in the daemon itself and we re-work the design to match this  
> consensus, re-aligning with our original proposal at NGCC.  
>  
> Apr 2018: Blake brings the discussion of repair scheduling to the dev list  
> (  
>  
> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E
>   
> ).  
> Many community members give positive feedback that we should solve it as  
> part of Cassandra and there is still no mention of contributing Reaper at  
> this point. The last message is my attempted summary giving context on how  
> we want to take the best of all the sidecars (OpsCenter, Priam, Reaper) and  
> ship them with Cassandra.  
>  
> Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design document  
> for gathering feedback on a general management sidecar. Sankalp and Dinesh  
> encourage Vinay and myself to kickstart that sidecar using the repair  
> scheduler patch  
>  
> Apr 2018: Dinesh reaches out to the dev list (  
>  
> https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E
>   
> )  
> about the general management process to gain further feedback. All feedback  
> remains positive as it is a potential place for multiple community members  
> to 

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

Right, I understand the arguments for starting a new project. I’m not saying 
reaper is, technically speaking, the best place to start. The point I’m trying 
to make is that the non-technical advantages of using an existing project as a 
starting point may outweigh the technical benefits of a clean slate. Whether 
that’s the case or not, it’s not a strictly technical decision, and the 
non-technical advantages of starting with reaper need to be weighed.

On September 7, 2018 at 8:19:50 PM, Jeff Jirsa (jji...@gmail.com) wrote:

The benefit is that it more closely matched the design doc, from 5 months ago, 
which is decidedly not about coordinating repair - it’s about a general purpose 
management tool, where repair is one of many proposed tasks  

https://docs.google.com/document/d/1UV9pE81NaIUF3g4L1wxq09nT11AkSQcMijgLFwGsY3s/edit
  


By starting with a tool that is built to run repair, you’re sacrificing 
generality and accepting something purpose built for one sub task. It’s an 
important subtask, and it’s a nice tool, but it’s not an implementation of the 
proposal, it’s an alternative that happens to do some of what was proposed.  

--  
Jeff Jirsa  


> On Sep 7, 2018, at 6:53 PM, Blake Eggleston  wrote:  
>  
> What’s the benefit of doing it that way vs starting with reaper and 
> integrating the netflix scheduler? If reaper was just a really inappropriate 
> choice for the cassandra management process, I could see that being a better 
> approach, but I don’t think that’s the case.  
>  
> If our management process isn’t a drop in replacement for reaper, then reaper 
> will continue to exist, which will split the user and developers base between 
> the 2 projects. That won't be good for either project.  
>  
> On September 7, 2018 at 6:12:01 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>  
> I’d also like to see the end state you describe: reaper UI wrapping the 
> Netflix management process with pluggable scheduling (either as is with 
> reaper now, or using the Netflix scheduler), but I don’t think that means we 
> need to start with reaper - if personally prefer the opposite direction, 
> starting with something small and isolated and layering on top.  
>  
> --  
> Jeff Jirsa  
>  
>  
>> On Sep 7, 2018, at 5:42 PM, Blake Eggleston  wrote:  
>>  
>> I think we should accept the reaper project as is and make that cassandra 
>> management process 1.0, then integrate the netflix scheduler (and other new 
>> features) into that.  
>>  
>> The ultimate goal would be for the netflix scheduler to become the default 
>> repair scheduler, but I think using reaper as the starting point makes it 
>> easier to get there.  
>>  
>> Reaper would bring a prod user base that would realistically take 2-3 years 
>> to build up with a new project. As an operator, switching to a cassandra 
>> management process that’s basically a re-brand of an existing and commonly 
>> used management process isn’t super risky. Asking operators to switch to a 
>> new process is a much harder sell.  
>>  
>> On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>>  
>> How can we continue moving this forward?  
>>  
>> Mick/Jon/TLP folks, is there a path here where we commit the  
>> Netflix-provided management process, and you augment Reaper to work with it? 
>>  
>> Is there a way we can make a larger umbrella that's modular that can  
>> support either/both?  
>> Does anyone believe there's a clear, objective argument that one is  
>> strictly better than the other? I haven't seen one.  
>>  
>>  
>>  
>> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
>>  wrote:  
>>  
>>> +1 to everything that Joey articulated with emphasis on the fact that  
>>> contributions should be evaluated based on the merit of code and their  
>>> value add to the whole offering. I hope it does not matter whether that  
>>> contribution comes from PMC member or a person who is not a committer. I  
>>> would like the process to be such that it encourages the new members to be  
>>> a part of the community and not shy away from contributing to the code  
>>> assuming their contributions are valued differently than committers or PMC  
>>> members. It would be sad to see the contributions decrease if we go down  
>>> that path.  
>>>  
>>> *Regards,*  
>>>  
>>> *Roopa Tangirala*  
>>>  
>>> Engineering Manager CDE  
>>>  
>>> *(408) 438-3156 - mobile*  
>>>  
>>>  
>>>  
>>>  
>>>  
>>>  
>>> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
>>> wrote: 

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

What’s the benefit of doing it that way vs starting with reaper and integrating 
the netflix scheduler? If reaper was just a really inappropriate choice for the 
cassandra management process, I could see that being a better approach, but I 
don’t think that’s the case.

If our management process isn’t a drop in replacement for reaper, then reaper 
will continue to exist, which will split the user and developers base between 
the 2 projects. That won't be good for either project.

On September 7, 2018 at 6:12:01 PM, Jeff Jirsa (jji...@gmail.com) wrote:

I’d also like to see the end state you describe: reaper UI wrapping the Netflix 
management process with pluggable scheduling (either as is with reaper now, or 
using the Netflix scheduler), but I don’t think that means we need to start 
with reaper - if personally prefer the opposite direction, starting with 
something small and isolated and layering on top.  

--  
Jeff Jirsa  


> On Sep 7, 2018, at 5:42 PM, Blake Eggleston  wrote:  
>  
> I think we should accept the reaper project as is and make that cassandra 
> management process 1.0, then integrate the netflix scheduler (and other new 
> features) into that.  
>  
> The ultimate goal would be for the netflix scheduler to become the default 
> repair scheduler, but I think using reaper as the starting point makes it 
> easier to get there.  
>  
> Reaper would bring a prod user base that would realistically take 2-3 years 
> to build up with a new project. As an operator, switching to a cassandra 
> management process that’s basically a re-brand of an existing and commonly 
> used management process isn’t super risky. Asking operators to switch to a 
> new process is a much harder sell.  
>  
> On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:  
>  
> How can we continue moving this forward?  
>  
> Mick/Jon/TLP folks, is there a path here where we commit the  
> Netflix-provided management process, and you augment Reaper to work with it?  
> Is there a way we can make a larger umbrella that's modular that can  
> support either/both?  
> Does anyone believe there's a clear, objective argument that one is  
> strictly better than the other? I haven't seen one.  
>  
>  
>  
> On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
>  wrote:  
>  
>> +1 to everything that Joey articulated with emphasis on the fact that  
>> contributions should be evaluated based on the merit of code and their  
>> value add to the whole offering. I hope it does not matter whether that  
>> contribution comes from PMC member or a person who is not a committer. I  
>> would like the process to be such that it encourages the new members to be  
>> a part of the community and not shy away from contributing to the code  
>> assuming their contributions are valued differently than committers or PMC  
>> members. It would be sad to see the contributions decrease if we go down  
>> that path.  
>>  
>> *Regards,*  
>>  
>> *Roopa Tangirala*  
>>  
>> Engineering Manager CDE  
>>  
>> *(408) 438-3156 - mobile*  
>>  
>>  
>>  
>>  
>>  
>>  
>> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
>> wrote:  
>>  
>>>> We are looking to contribute Reaper to the Cassandra project.  
>>>>  
>>> Just to clarify are you proposing contributing Reaper as a project via  
>>> donation or you are planning on contributing the features of Reaper as  
>>> patches to Cassandra? If the former how far along are you on the donation  
>>> process? If the latter, when do you think you would have patches ready  
>> for  
>>> consideration / review?  
>>>  
>>>  
>>>> Looking at the patch it's very similar in its base design already, but  
>>>> Reaper does has a lot more to offer. We have all been working hard to  
>>> move  
>>>> it to also being a side-car so it can be contributed. This raises a  
>>> number  
>>>> of relevant questions to this thread: would we then accept both works  
>> in  
>>>> the Cassandra project, and what burden would it put on the current PMC  
>> to  
>>>> maintain both works.  
>>>>  
>>> I would hope that we would collaborate on merging the best parts of all  
>>> into the official Cassandra sidecar, taking the always on, shared  
>> nothing,  
>>> highly available system that we've contributed a patchset for and adding  
>> in  
>>> many of the repair features (e.g. schedules, a nice web UI) that Reaper  
>>> has.  
>>>  
>>>  
>>>> I share Stefan's conce

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

I think we should accept the reaper project as is and make that cassandra 
management process 1.0, then integrate the netflix scheduler (and other new 
features) into that.

The ultimate goal would be for the netflix scheduler to become the default 
repair scheduler, but I think using reaper as the starting point makes it 
easier to get there. 

Reaper would bring a prod user base that would realistically take 2-3 years to 
build up with a new project. As an operator, switching to a cassandra 
management process that’s basically a re-brand of an existing and commonly used 
management process isn’t super risky. Asking operators to switch to a new 
process is a much harder sell. 

On September 7, 2018 at 4:17:10 PM, Jeff Jirsa (jji...@gmail.com) wrote:

How can we continue moving this forward?  

Mick/Jon/TLP folks, is there a path here where we commit the  
Netflix-provided management process, and you augment Reaper to work with it?  
Is there a way we can make a larger umbrella that's modular that can  
support either/both?  
Does anyone believe there's a clear, objective argument that one is  
strictly better than the other? I haven't seen one.  



On Mon, Aug 20, 2018 at 4:14 PM Roopa Tangirala  
 wrote:  

> +1 to everything that Joey articulated with emphasis on the fact that  
> contributions should be evaluated based on the merit of code and their  
> value add to the whole offering. I hope it does not matter whether that  
> contribution comes from PMC member or a person who is not a committer. I  
> would like the process to be such that it encourages the new members to be  
> a part of the community and not shy away from contributing to the code  
> assuming their contributions are valued differently than committers or PMC  
> members. It would be sad to see the contributions decrease if we go down  
> that path.  
>  
> *Regards,*  
>  
> *Roopa Tangirala*  
>  
> Engineering Manager CDE  
>  
> *(408) 438-3156 - mobile*  
>  
>  
>  
>  
>  
>  
> On Mon, Aug 20, 2018 at 2:58 PM Joseph Lynch   
> wrote:  
>  
> > > We are looking to contribute Reaper to the Cassandra project.  
> > >  
> > Just to clarify are you proposing contributing Reaper as a project via  
> > donation or you are planning on contributing the features of Reaper as  
> > patches to Cassandra? If the former how far along are you on the donation  
> > process? If the latter, when do you think you would have patches ready  
> for  
> > consideration / review?  
> >  
> >  
> > > Looking at the patch it's very similar in its base design already, but  
> > > Reaper does has a lot more to offer. We have all been working hard to  
> > move  
> > > it to also being a side-car so it can be contributed. This raises a  
> > number  
> > > of relevant questions to this thread: would we then accept both works  
> in  
> > > the Cassandra project, and what burden would it put on the current PMC  
> to  
> > > maintain both works.  
> > >  
> > I would hope that we would collaborate on merging the best parts of all  
> > into the official Cassandra sidecar, taking the always on, shared  
> nothing,  
> > highly available system that we've contributed a patchset for and adding  
> in  
> > many of the repair features (e.g. schedules, a nice web UI) that Reaper  
> > has.  
> >  
> >  
> > > I share Stefan's concern that consensus had not been met around a  
> > > side-car, and that it was somehow default accepted before a patch  
> landed.  
> >  
> >  
> > I feel this is not correct or fair. The sidecar and repair discussions  
> have  
> > been anything _but_ "default accepted". The timeline of consensus  
> building  
> > involving the management sidecar and repair scheduling plans:  
> >  
> > Dec 2016: Vinay worked with Jon and Alex to try to collaborate on Reaper  
> to  
> > come up with design goals for a repair scheduler that could work at  
> Netflix  
> > scale.  
> >  
> > ~Feb 2017: Netflix believes that the fundamental design gaps prevented us  
> > from using Reaper as it relies heavily on remote JMX connections and  
> > central coordination.  
> >  
> > Sep. 2017: Vinay gives a lightning talk at NGCC about a highly available  
> > and distributed repair scheduling sidecar/tool. He is encouraged by  
> > multiple committers to build repair scheduling into the daemon itself and  
> > not as a sidecar so the database is truly eventually consistent.  
> >  
> > ~Jun. 2017 - Feb. 2018: Based on internal need and the positive feedback  
> at  
> > NGCC, Vinay and myself prototype the distributed repair scheduler within  
> > Priam and roll it out at Netflix scale.  
> >  
> > Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 page  
> > design document for adding repair scheduling to the daemon itself and  
> open  
> > the design up for feedback from the community. We get feedback from Alex,  
> > Blake, Nate, Stefan, and Mick. As far as I know there were zero proposals  
> > to contribute Reaper at this point. 

Re: Reaper as cassandra-admin

[Blake Eggleston]

> FTR nobody has called Reaper a "hopeless mess".

I didn't mean they did. I just meant that it's generally a bad idea to do a 
rewrite unless the thing being rewritten is a hopeless mess, which reaper 
probably isn't. I realize this isn't technically a rewrite since we're not 
talking about actually rewriting something that's part of the project, but a 
lot of the same reasoning applies to starting work on a new admin tool vs using 
reaper as a starting point. It's not a strictly technical decision either. The 
community of users and developers already established around reaper is also a 
consideration.
On August 28, 2018 at 3:53:02 PM, dinesh.jo...@yahoo.com.INVALID 
(dinesh.jo...@yahoo.com.invalid) wrote:

On Tuesday, August 28, 2018, 2:52:03 PM PDT, Blake Eggleston 
 wrote:  
> I’m sure reaper will bring tech debt with it, but I doubt it's a hopeless 
> mess.   
FTR nobody has called Reaper a "hopeless mess".  
> It would bring a relatively mature project as well as a community of users> 
> and developers that the other options won’t. It’s probably a lot less work to 
> > rework whatever shortcomings reaper has, add new-hotness repair  

You can bring in parts of a relatively mature project that minimize refactoring 
& changes that need to be made once imported. You can also bring in best parts 
of multiples projects without importing entire codebases.  
Dinesh  


On August 28, 2018 at 1:40:59 PM, Roopa Tangirala 
(rtangir...@netflix.com.invalid) wrote:  
I share Dinesh's concern too regarding tech debt with existing codebase.   
Its good we have multiple solutions for repairs which have been always   
painful in Cassandra. It would be great to see the community take the best   
pieces from the available solutions and roll it into the fresh side car   
which will help ease Cassandra's maintenance for lot of folks.   

My main concern with starting with an existing codebase is that it comes   
with tech debt. This is not specific to Reaper but to any codebase that is   
imported as a whole. This means future developers and patches have to work   
within the confines of the decisions that were already made. Practically   
speaking once a codebase is established there is inertia in making   
architectural changes and we're left dealing with technical debt.   



*Regards,*   

*Roopa Tangirala*   

Engineering Manager CDE   

*(408) 438-3156 - mobile*   






On Mon, Aug 27, 2018 at 10:49 PM Dinesh Joshi   
 wrote:   

> > On Aug 27, 2018, at 5:36 PM, Jonathan Haddad  wrote:   
> > We're hoping to get some feedback on our side if that's something people   
> > are interested in. We've gone back and forth privately on our own   
> > preferences, hopes, dreams, etc, but I feel like a public discussion   
> would   
> > be healthy at this point. Does anyone share the view of using Reaper as   
> a   
> > starting point? What concerns to people have?   
>   
>   
> I have briefly looked at the Reaper codebase but I am yet to analyze it   
> better to have a real, meaningful opinion.   
>   
> My main concern with starting with an existing codebase is that it comes   
> with tech debt. This is not specific to Reaper but to any codebase that is   
> imported as a whole. This means future developers and patches have to work   
> within the confines of the decisions that were already made. Practically   
> speaking once a codebase is established there is inertia in making   
> architectural changes and we're left dealing with technical debt.   
>   
> As it stands I am not against the idea of using Reaper's features and I   
> would very much like using mature code that has been tested. I would   
> however like to propose piece-mealing it into the codebase. This will give   
> the community a chance to review what is going in and possibly change some   
> of the design decisions upfront. This will also avoid a situation where we   
> have to make many breaking changes in the initial versions due to   
> refactoring.   
>   
> I would also like it if we could compare and contrast the functionality   
> with Priam or any other interesting sidecars that folks may want to call   
> out. In fact it would be great if we could bring in the best functionality   
> from multiple implementations.   
>   
> Dinesh   
> -   
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org   
> For additional commands, e-mail: dev-h...@cassandra.apache.org   
>   
> 

Re: Reaper as cassandra-admin

[Blake Eggleston]

I haven’t settled on a position yet (will have more time think about things 
after the 9/1 freeze), but I wanted to point out that the argument that 
something new should be written because an existing project has tech debt, and 
we'll do it the right way this time, is a pretty common software engineering 
mistake. The thing you’re replacing usually needs to have some really serious 
problems to make it worth replacing.

I’m sure reaper will bring tech debt with it, but I doubt it's a hopeless mess. 
It would bring a relatively mature project as well as a community of users and 
developers that the other options won’t. It’s probably a lot less work to 
rework whatever shortcomings reaper has, add new-hotness repair schedulers to 
it, and get people to actually use them than it would be to write something 
from scratch and build community confidence in it and get reaper users to 
switch.

On August 28, 2018 at 1:40:59 PM, Roopa Tangirala 
(rtangir...@netflix.com.invalid) wrote:
I share Dinesh's concern too regarding tech debt with existing codebase.  
Its good we have multiple solutions for repairs which have been always  
painful in Cassandra. It would be great to see the community take the best  
pieces from the available solutions and roll it into the fresh side car  
which will help ease Cassandra's maintenance for lot of folks.  

My main concern with starting with an existing codebase is that it comes  
with tech debt. This is not specific to Reaper but to any codebase that is  
imported as a whole. This means future developers and patches have to work  
within the confines of the decisions that were already made. Practically  
speaking once a codebase is established there is inertia in making  
architectural changes and we're left dealing with technical debt.  



*Regards,*  

*Roopa Tangirala*  

Engineering Manager CDE  

*(408) 438-3156 - mobile*  






On Mon, Aug 27, 2018 at 10:49 PM Dinesh Joshi  
 wrote:  

> > On Aug 27, 2018, at 5:36 PM, Jonathan Haddad  wrote:  
> > We're hoping to get some feedback on our side if that's something people  
> > are interested in. We've gone back and forth privately on our own  
> > preferences, hopes, dreams, etc, but I feel like a public discussion  
> would  
> > be healthy at this point. Does anyone share the view of using Reaper as  
> a  
> > starting point? What concerns to people have?  
>  
>  
> I have briefly looked at the Reaper codebase but I am yet to analyze it  
> better to have a real, meaningful opinion.  
>  
> My main concern with starting with an existing codebase is that it comes  
> with tech debt. This is not specific to Reaper but to any codebase that is  
> imported as a whole. This means future developers and patches have to work  
> within the confines of the decisions that were already made. Practically  
> speaking once a codebase is established there is inertia in making  
> architectural changes and we're left dealing with technical debt.  
>  
> As it stands I am not against the idea of using Reaper's features and I  
> would very much like using mature code that has been tested. I would  
> however like to propose piece-mealing it into the codebase. This will give  
> the community a chance to review what is going in and possibly change some  
> of the design decisions upfront. This will also avoid a situation where we  
> have to make many breaking changes in the initial versions due to  
> refactoring.  
>  
> I would also like it if we could compare and contrast the functionality  
> with Priam or any other interesting sidecars that folks may want to call  
> out. In fact it would be great if we could bring in the best functionality  
> from multiple implementations.  
>  
> Dinesh  
> -  
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
> For additional commands, e-mail: dev-h...@cassandra.apache.org  
>  
>  

Re: Proposing an Apache Cassandra Management process

[Blake Eggleston]

I'd be more in favor of making it a separate project, basically for all the 
reasons listed below. I'm assuming we'd want a management process to work 
across different versions, which will be more awkward if it's in tree. Even if 
that's not the case, keeping it in a different repo at this point will make 
iteration easier than if it were in tree. I'd imagine (or at least hope) that 
validating the management process for release would be less difficult than the 
main project, so tying them to the Cassandra release cycle seems unnecessarily 
restrictive.


On August 17, 2018 at 12:07:18 AM, Dinesh Joshi 
(dinesh.jo...@yahoo.com.invalid) wrote:

> On Aug 16, 2018, at 9:27 PM, Sankalp Kohli  wrote: 
> 
> I am bumping this thread because patch has landed for this with repair 
> functionality. 
> 
> I have a following proposal for this which I can put in the JIRA or doc 
> 
> 1. We should see if we can keep this in a separate repo like Dtest. 

This would imply a looser coupling between the two. Keeping things in-tree is 
my preferred approach. It makes testing, dependency management and code sharing 
easier. 

> 2. It should have its own release process. 

This means now there would be two releases that need to be managed and 
coordinated. 

> 3. It should have integration tests for different versions of Cassandra it 
> will support. 

Given the lack of test infrastructure - this will be hard especially if you 
have to qualify a matrix of builds. 

Dinesh 
- 
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
For additional commands, e-mail: dev-h...@cassandra.apache.org 

Re: [VOTE] Release Apache Cassandra 3.0.17 (Take 2)

[Blake Eggleston]

+1


On July 26, 2018 at 9:27:11 PM, Marcus Eriksson (krum...@gmail.com) wrote:

+1 

On Fri, Jul 27, 2018 at 5:03 AM Vinay Chella  
wrote: 

> +1 nb. 
> 
> Here are the test results. 
> https://circleci.com/gh/vinaykumarchella/cassandra/tree/3.0.17_tentative 
> 
> Most of the failed tests are related to snapshot_test.TestArchiveCommitlog. 
> 
> Regards, 
> Vinay Chella 
> 
> 
> On Thu, Jul 26, 2018 at 3:05 PM kurt greaves  wrote: 
> 
> > +1 nb 
> > 
> > On Fri., 27 Jul. 2018, 00:20 Sam Tunnicliffe,  wrote: 
> > 
> > > +1 
> > > 
> > > On 25 July 2018 at 08:17, Michael Shuler  
> wrote: 
> > > 
> > > > I propose the following artifacts for release as 3.0.17. 
> > > > 
> > > > sha1: d52c7b8c595cc0d06fc3607bf16e3f595f016bb6 
> > > > Git: 
> > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > > shortlog;h=refs/tags/3.0.17-tentative 
> > > > Artifacts: 
> > > > https://repository.apache.org/content/repositories/ 
> > > > orgapachecassandra-1165/org/apache/cassandra/apache-cassandra/3.0.17/ 
> > > > Staging repository: 
> > > > https://repository.apache.org/content/repositories/ 
> > > > orgapachecassandra-1165/ 
> > > > 
> > > > The Debian and RPM packages are available here: 
> > > > http://people.apache.org/~mshuler 
> > > > 
> > > > The vote will be open for 72 hours (longer if needed). 
> > > > 
> > > > [1]: CHANGES.txt: 
> > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > > blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.17-tentative 
> > > > [2]: NEWS.txt: 
> > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > > blob_plain;f=CHANGES.txt;hb=refs/tags/3.0.17-tentative 
> > > > 
> > > > - 
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org 
> > > > 
> > > > 
> > > 
> > 
> 

Re: [VOTE] Release Apache Cassandra 2.2.13

[Blake Eggleston]

+1

On July 26, 2018 at 9:26:48 PM, Marcus Eriksson (krum...@gmail.com) wrote:

+1 

On Fri, Jul 27, 2018 at 12:05 AM kurt greaves  wrote: 

> +1 nb 
> 
> On Fri., 27 Jul. 2018, 00:20 Sam Tunnicliffe,  wrote: 
> 
> > +1 
> > 
> > On 25 July 2018 at 08:17, Michael Shuler  wrote: 
> > 
> > > I propose the following artifacts for release as 2.2.13. 
> > > 
> > > sha1: 3482370df5672c9337a16a8a52baba53b70a4fe8 
> > > Git: 
> > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > shortlog;h=refs/tags/2.2.13-tentative 
> > > Artifacts: 
> > > https://repository.apache.org/content/repositories/ 
> > > orgapachecassandra-1167/org/apache/cassandra/apache-cassandra/2.2.13/ 
> > > Staging repository: 
> > > https://repository.apache.org/content/repositories/ 
> > > orgapachecassandra-1167/ 
> > > 
> > > The Debian and RPM packages are available here: 
> > > http://people.apache.org/~mshuler 
> > > 
> > > The vote will be open for 72 hours (longer if needed). 
> > > 
> > > [1]: CHANGES.txt: 
> > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.13-tentative 
> > > [2]: NEWS.txt: 
> > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > blob_plain;f=CHANGES.txt;hb=refs/tags/2.2.13-tentative 
> > > 
> > > - 
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org 
> > > 
> > > 
> > 
> 

Re: [VOTE] Release Apache Cassandra 3.11.3 (Take 2)

[Blake Eggleston]

+1


On July 26, 2018 at 9:27:27 PM, Marcus Eriksson (krum...@gmail.com) wrote:

+1 

On Fri, Jul 27, 2018 at 4:59 AM Vinay Chella  
wrote: 

> +1 nb. 
> 
> Here are the failed tests (circleci 
> < 
> https://circleci.com/gh/vinaykumarchella/workflows/cassandra/tree/3.11.3_tentative
>  
> >), 
> if anyone is curious about failed tests. 
> 
> dtests-no-vnodes (5 Failed tests) 
> test_short_read - consistency_test.TestConsistency 
> test_describecluster_more_information_three_datacenters - 
> nodetool_test.TestNodetool 
> test_failure_threshold_deletions - paging_test.TestPagingWithDeletions 
> test_closing_connections - thrift_hsha_test.TestThriftHSHA 
> test_mutation_v5 - write_failures_test.TestWriteFailures 
> 
> dtests-with-vnodes (6 failed tests) 
> test_14330 - consistency_test.TestConsistency 
> test_remote_query - cql_test.TestCQLSlowQuery 
> test_describecluster_more_information_three_datacenters - 
> nodetool_test.TestNodetool 
> test_failure_threshold_deletions - paging_test.TestPagingWithDeletions 
> test_closing_connections - thrift_hsha_test.TestThriftHSHA 
> test_mutation_v5 - write_failures_test.TestWriteFailures 
> 
> Regards, 
> Vinay Chella 
> 
> 
> On Thu, Jul 26, 2018 at 3:06 PM kurt greaves  wrote: 
> 
> > +1 nb 
> > 
> > On Fri., 27 Jul. 2018, 00:20 Sam Tunnicliffe,  wrote: 
> > 
> > > +1 
> > > 
> > > On 25 July 2018 at 08:16, Michael Shuler  
> wrote: 
> > > 
> > > > I propose the following artifacts for release as 3.11.3. 
> > > > 
> > > > sha1: 31d5d870f9f5b56391db46ba6cdf9e0882d8a5c0 
> > > > Git: 
> > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > > shortlog;h=refs/tags/3.11.3-tentative 
> > > > Artifacts: 
> > > > https://repository.apache.org/content/repositories/ 
> > > > orgapachecassandra-1164/org/apache/cassandra/apache-cassandra/3.11.3/ 
> > > > Staging repository: 
> > > > https://repository.apache.org/content/repositories/ 
> > > > orgapachecassandra-1164/ 
> > > > 
> > > > The Debian and RPM packages are available here: 
> > > > http://people.apache.org/~mshuler 
> > > > 
> > > > The vote will be open for 72 hours (longer if needed). 
> > > > 
> > > > [1]: CHANGES.txt: 
> > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > > blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.3-tentative 
> > > > [2]: NEWS.txt: 
> > > > http://git-wip-us.apache.org/repos/asf?p=cassandra.git;a= 
> > > > blob_plain;f=CHANGES.txt;hb=refs/tags/3.11.3-tentative 
> > > > 
> > > > - 
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org 
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org 
> > > > 
> > > > 
> > > 
> > 
> 

Re: In need of reviewers

[Blake Eggleston]

I'll spend a day or two working through some of these next week.

On 5/11/18, 3:44 AM, "kurt greaves"  wrote:

We've got a bunch of tickets that are either in need of review or just a
bit of feedback. Would be very grateful for any help here :).

Bugs:
https://issues.apache.org/jira/browse/CASSANDRA-14365
https://issues.apache.org/jira/browse/CASSANDRA-14204
https://issues.apache.org/jira/browse/CASSANDRA-14162
https://issues.apache.org/jira/browse/CASSANDRA-14126
https://issues.apache.org/jira/browse/CASSANDRA-14365
https://issues.apache.org/jira/browse/CASSANDRA-14099
https://issues.apache.org/jira/browse/CASSANDRA-14073
https://issues.apache.org/jira/browse/CASSANDRA-14063
https://issues.apache.org/jira/browse/CASSANDRA-14056
https://issues.apache.org/jira/browse/CASSANDRA-14054
https://issues.apache.org/jira/browse/CASSANDRA-14013
https://issues.apache.org/jira/browse/CASSANDRA-13841
https://issues.apache.org/jira/browse/CASSANDRA-13698

Improvements:
https://issues.apache.org/jira/browse/CASSANDRA-14309
https://issues.apache.org/jira/browse/CASSANDRA-10789
https://issues.apache.org/jira/browse/CASSANDRA-14443
https://issues.apache.org/jira/browse/CASSANDRA-13010
https://issues.apache.org/jira/browse/CASSANDRA-11559
https://issues.apache.org/jira/browse/CASSANDRA-10789
https://issues.apache.org/jira/browse/CASSANDRA-10023
https://issues.apache.org/jira/browse/CASSANDRA-8460

Cheers,
Kurt




-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Repair scheduling tools

[Blake Eggleston]

h information to run subrange repairs across multiple keyspaces
>> or
>> > > even non-overlapping ranges at the same time. That lets people
>> experiment
>> > > with and quickly/safely/easily iterate on different scheduling
>> strategies
>> > > in the short term, and long-term those strategies can be integrated
>> into a
>> > > built-in scheduler
>> > >
>> > > On the subject of scheduling, I think adjusting
>> parallelism/aggression with
>> > > a possible whitelist or blacklist would be a lot more useful than a
>> "time
>> > > between repairs". That is, if repairs run for a few hours then don't
>> run
>> > > for a few (somewhat hard-to-predict) hours, I still have to size the
>> > > cluster for the load when the repairs are running. The only reason I
>> can
>> > > think of for an interval between repairs is to allow re-compaction
>> from
>> > > repair anticompactions, and subrange repairs seem to eliminate this.
>> Even
>> > > if they didn't, a more direct method along the lines of "don't repair
>> when
>> > > the compaction queue is too long" might make more sense. Blacklisted
    >> > > timeslots might be useful for avoiding peak time or batch jobs, but
>> only if
>> > > they can be specified for consistent time-of-day intervals instead of
>> > > unpredictable lulls between repairs.
>> > >
>> > > I really like the idea of automatically adjusting gc_grace_seconds
>> based on
>> > > repair state. The only_purge_repaired_tombstones option fixes this
>> > > elegantly for sequential/incremental repairs on STCS, but not for
>> subrange
>> > > repairs or LCS (unless a scheduler gains the ability somehow to
>> determine
>> > > that every subrange in an sstable has been repaired and mark it
>> > > accordingly?)
>> > >
>> > >
>> > > On 2018/04/03 17:48:14, Blake Eggleston <b...@apple.com> wrote:
>> > > > Hi dev@,
>> > > >
>> > > > >
>> > > >
>> > > > The question of the best way to schedule repairs came up on
>> > > CASSANDRA-14346, and I thought it would be good to bring up the idea
>> of an
>> > > external tool on the dev list.
>> > > >
>> > > > >
>> > > >
>> > > > Cassandra lacks any sort of tools for automating routine tasks that
>> are
>> > > required for running clusters, specifically repair. Regular repair is
>> a
>> > > must for most clusters, like compaction. This means that, especially
>> as far
>> > > as eventual consistency is concerned, Cassandra isn’t totally
>> functional
>> > > out of the box. Operators either need to find a 3rd party solution or
>> > > implement one themselves. Adding this to Cassandra would make it
>> easier to
>> > > use.
>> > > >
>> > > > >
>> > > >
>> > > > Is this something we should be doing? If so, what should it look
>> like?
>> > > >
>> > > > >
>> > > >
>> > > > Personally, I feel like this is a pretty big gap in the project and
>> would
>> > > like to see an out of process tool offered. Ideally, Cassandra would
>> just
>> > > take care of itself, but writing a distributed repair scheduler that
>> you
>> > > trust to run in production is a lot harder than writing a single
>> process
>> > > management application that can failover.
>> > > >
>> > > > >
>> > > >
>> > > > Any thoughts on this?
>> > > >
>> > > > >
>> > > >
>> > > > Thanks,
>> > > >
>> > > > >
>> > > >
>> > > > Blake
>> > > >
>> > > >
>> > >
>>
>
>




-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Roadmap for 4.0

[Blake Eggleston]

I agree that not releasing semi-regularly is not good for the project. I think 
our habit of releasing half working software is much worse though. Our 
testing/stability story is not iron clad. I really think the bar for releasing 
4.0 should be that the people in this thread are running the code in 
production, recommending their customers run it in production, or offering and 
supporting it as part of their cloud service.

In that context, the argument for waiting for some features is less about 
trying to do all the things and more about making 4.0 something worth the time 
and expense of validating for production.

On 4/11/18, 1:06 AM, "Sylvain Lebresne"  wrote:

On Wed, Apr 11, 2018 at 12:35 AM Jeff Jirsa  wrote:

> Seriously, what's the rush to branch? Do we all love merging so much we
> want to do a few more times just for the sake of merging? If nothing
> diverges, there's nothing gained from the branch, and if it did diverge, 
we
> add work for no real gain.
>

Again, to me, the "rush" is that 1) there is tons of changes sitting in
trunk
that some user (_not all_, granted)[1], especially new ones, would likely
benefits, and sooner for those is better than later, 2) we want to favor
release stability and we *know* from years of experience (and frankly,
common
sense) that the bigger the release is, the harder it is to test it/ensuring
overall stability[2] and 3) not having major releases for years[3] is
impacting at least the perceived dynamism/liveness of the project to
external
actors (prospective new user come in mind here, but not only) and that's
simply bad for the project.

And having listed arguments for a soon freeze/not accumulating much more
before release, I'd like to reverse the question to you: what are the big
downsides of not doing that? Are we really that hung up on our own
developers
comfort that the annoyance of a bit more merging trumps the arguments above?

Anyway, the reasons above make me thing that it's better _for the project_
to freeze 4.0 soon, which doesn't exclude a "short" cycle for the following
major (where my definition of short here is something like 6-8 months), and
I'm happy to decide to make 4.0 a non-mandatory upgrade to whatever
comes next so that folks that prefer upgrading rarely can simply skip it and
go to the next one. Likely nobody will die if we wait more though, and it's
clear it will make a few people here more happy if we do, but I believe the
project as a whole will be a bit worst off, that's all.

--
Sylvain


[1]: I'll note that I don't deny upgrading is huge deal for some users, but
let's not skew arguments too much based on any one user interest. For many
users, upgrading even every year to get improvements is still considered as
a
good deal, and that's not counting new users for which it's super
frustrating
to miss out on improvements because we release major only every 2+ years.
[2]: I'll be clear: I will simply not buy anyone argument that "we'll do
so much better testing this time" on face value. Not anymore. If you want to
use that argument to sell having bigger releases, then prove it first. Let's
do reasonably sized 4.0 and 4.1/5.0 and prove that our testing/stability
story
is iron clad now, and then for 4.2/6.0 I'll be willing to agree that making
bigger release may not impact stability too much.
[3]: Conservative estimate, if we do care about stable releases as we all
seem
to, even if we were to freeze June 1, we will almost surely not release
before
October/November, which will be ~1.3 year since the last major release
(again,
that's the conservative estimate). If we push a few months to get some big
complex feature in, not only this push the freeze of those few months, but
will also require more testing, so we're looking at 2+ years, with a
possibly
large '+'.




>
> Beyond that, I still don't like June 1. Validating releases is hard. It
> sounds easy to drop a 4.1 and ask people to validate again, but it's a 
hell
> of a lot harder than it sounds. I'm not saying I'm a hard -1, but I really
> think it's too soon. 50'ish days is too short to draw a line in the sand,
> especially as people balance work obligations with Cassandra feature
> development.
>
>
>
>
> On Tue, Apr 10, 2018 at 3:18 PM, Nate McCall  wrote:
>
> > A lot of good points and everyone's input is really appreciated.
> >
> > So it sounds like we are building consensus towards June 1 for 4.0
> > branch point/feature freeze and the goal is stability. (No one has
> > come with a hard NO anyway).
> >
> > I want to reiterate Sylvain's point that we can do whatever we 

Re: Roadmap for 4.0

[Blake Eggleston]

+1

On 4/4/18, 5:48 PM, "Jeff Jirsa"  wrote:

Earlier than I’d have personally picked, but I’m +1 too



-- 
Jeff Jirsa


> On Apr 4, 2018, at 5:06 PM, Nate McCall  wrote:
> 
> Top-posting as I think this summary is on point - thanks, Scott! (And
> great to have you back, btw).
> 
> It feels to me like we are coalescing on two points:
> 1. June 1 as a freeze for alpha
> 2. "Stable" is the new "Exciting" (and the testing and dogfooding
> implied by such before a GA)
> 
> How do folks feel about the above points?
> 
> 
>> Re-raising a point made earlier in the thread by Jeff and affirmed by 
Josh:
>> 
>> –––
>> Jeff:
 A hard date for a feature freeze makes sense, a hard date for a release
 does not.
>> 
>> Josh:
>>> Strongly agree. We should also collectively define what "Done" looks 
like
>>> post freeze so we don't end up in bike-shedding hell like we have in the
>>> past.
>> –––
>> 
>> Another way of saying this: ensuring that the 4.0 release is of high 
quality is more important than cutting the release on a specific date.
>> 
>> If we adopt Sylvain's suggestion of freezing features on a "feature 
complete" date (modulo a "definition of done" as Josh suggested), that will 
help us align toward the polish, performance work, and dog-fooding needed to 
feel great about shipping 4.0. It's a good time to start thinking about the 
approaches to testing, profiling, and dog-fooding various contributors will 
want to take on before release.
>> 
>> I love how Ben put it:
>> 
>>> An "exciting" 4.0 release to me is one that is stable and usable
>>> with no perf regressions on day 1 and includes some of the big
>>> internal changes mentioned previously.
>>> 
>>> This will set the community up well for some awesome and exciting
>>> stuff that will still be in the pipeline if it doesn't make it to 4.0.
>> 
>> That sounds great to me, too.
>> 
>> – Scott
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org





-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Repair scheduling tools

[Blake Eggleston]

Hi dev@,

 

The question of the best way to schedule repairs came up on CASSANDRA-14346, 
and I thought it would be good to bring up the idea of an external tool on the 
dev list.

 

Cassandra lacks any sort of tools for automating routine tasks that are 
required for running clusters, specifically repair. Regular repair is a must 
for most clusters, like compaction. This means that, especially as far as 
eventual consistency is concerned, Cassandra isn’t totally functional out of 
the box. Operators either need to find a 3rd party solution or implement one 
themselves. Adding this to Cassandra would make it easier to use.

 

Is this something we should be doing? If so, what should it look like?

 

Personally, I feel like this is a pretty big gap in the project and would like 
to see an out of process tool offered. Ideally, Cassandra would just take care 
of itself, but writing a distributed repair scheduler that you trust to run in 
production is a lot harder than writing a single process management application 
that can failover.

 

Any thoughts on this?

 

Thanks,

 

Blake

Re: A JIRA proposing a seperate repository for the online documentation

[Blake Eggleston]

It would probably be more productive to list some specific concerns you have 
with Hugo. Then explain why you think they make using it a bad idea. Then offer 
some alternatives.

On 3/16/18, 1:18 PM, "Kenneth Brotman"  wrote:

Thanks for that Eric Evans.  

I'm not sure Hugo is the way to go.  I don't see how I would generate the 
quality of work I would want with it.  It seems like another example of coders 
learning and using a more complicated program to generate the code they could 
have already generated - it’s a disease in the I.T. industry right now.  But I 
could be wrong.  

Here's the thing.  I've been spending a lot of my time for the past three 
weeks now trying to help with the website.  That is a tiny website.  I've never 
worked with a website that tiny.  Bear with me.  

I'm studying Jeff Carpenter and Eben Hewitt's book: Cassandra The 
Definitive Guide 
https://www.amazon.com/Cassandra-Definitive-Guide-Distributed-Scale/dp/1491933666/ref=sr_1_1?ie=UTF8=1521230539=8-1=cassandra+the+definitive+guide
 and have already have a terrible itch to start contributing some code.  I just 
want to get set up to do that.  The book seems to be a good way to get familiar 
with the internals and the code of Cassandra.

I can only do so much for the group at one time just like anyone else.  
I'll only do top quality work.  I'll only be a part of top quality work.  It 
could be that I won't feel comfortable with what the group wants to do for the 
website.  

Please keep working on it as it is really embarrassing, terrible, 
substandard unacceptable beneath professional standards...

I will contribute if it's possible for me to do so. Let's see what we 
decide to do going forward for the website.

Kenneth Brotman
(Cassandra coder?) 

-Original Message-
From: Eric Evans [mailto:john.eric.ev...@gmail.com] 
Sent: Friday, March 16, 2018 7:59 AM
To: dev@cassandra.apache.org
Subject: Re: A JIRA proposing a seperate repository for the online 
documentation

On Thu, Mar 15, 2018 at 11:40 AM, Kenneth Brotman 
 wrote:
> Well pickle my cucumbers Jon!  It's good to know that you have experience 
with Hugo, see it as a good fit and that all has been well.  I look forward to 
the jira epic!
>
> How exactly does the group make such a decision:  Call for final 
discussion?  Call for vote?  Wait for the PMC to vote?

Good question!

Decisions like this are made by consensus; As the person who is attempting 
to do something, you discuss your ideas with the group, listen to the feedback 
of others, and develop consensus around a direction.

This is more difficult than demanding your way, or having someone(s) in a 
position of absolute power tell you what you can and cannot do, but the result 
is better.


> -Original Message-
> From: Jon Haddad [mailto:jonathan.had...@gmail.com] On Behalf Of Jon 
> Haddad
> Sent: Thursday, March 15, 2018 9:24 AM
> To: dev@cassandra.apache.org
> Subject: Re: A JIRA proposing a seperate repository for the online 
> documentation
>
> Murukesh is correct on a very useable, pretty standard process of 
multi-versioned docs.
>
> I’ll put my thoughts in a JIRA epic tonight.  I’ll be a multi-phase 
process.  Also correct in that I’d like us to move to Hugo for the site, I’d 
like us to have a unified system between the site & the docs, and hugo has been 
excellent. We run the reaper site & docs off hugo, it works well.  We just 
don’t do multi-versions (because we don’t support multiple): 
https://github.com/thelastpickle/cassandra-reaper/tree/master/src/docs 
.
>
> Jon
>
>> On Mar 15, 2018, at 8:57 AM, Murukesh Mohanan 
 wrote:
>>
>> On Fri, Mar 16, 2018 at 0:19 Kenneth Brotman 
>> >
>> wrote:
>>
>>> Help me out here.  I could have had a website with support for more 
>>> than one version done several different ways by now.
>>>
>>> A website with several versions of documentation is going to have 
>>> sub-directories for each version of documentation obviously.  I've 
>>> offered to create those sub-directories under the "doc" folder of 
>>> the current repository; and I've offered to move the online 
>>> documentation to a separate repository and have the sub-directories 
>>> there.  Both were shot down.  Is there a third way?  If so please just 
spill the beans.
>>>
>>
>> There is. Note that the website is an independent repository. So to 
>> host docs for multiple versions, only the website's repository (or 
>> rather, the final built contents) needs multiple directories. You can 
>> 

Re: Expensive metrics?

[Blake Eggleston]

Hi Micke,

This is really cool, thanks for taking the time to investigate this. I believe 
the metrics around memtable insert time come in handy in identifying high 
partition contention in the memtable. I know I've been involved in a situation 
over the past year where we got actionable info from this metric. Reducing 
resolution to milliseconds is probably a no go since most things in this path 
should complete in less than a millisecond. 

Revisiting the use of the codahale metrics in the hot path like this definitely 
seems like a good idea though. I don't think it's been something we've talked 
about a lot, and it definitely looks like we could benefit from using something 
more specialized here. I think it's worth doing, especially since there won't 
be any major changes to how we do threading in 4.0. It's probably also worth 
opening a JIRA and investigating the calls to nano time. We at least need 
microsecond resolution here, and there could be something we haven't thought 
of? It's worth a look at least.

Thanks,

Blake

On 2/22/18, 6:10 AM, "Michael Burman"  wrote:

Hi,

I wanted to get some input from the mailing list before making a JIRA 
and potential fixes. I'll touch the performance more on latter part, but 
there's one important question regarding the write latency metric 
recording place. Currently we measure the writeLatency (and metric write 
sampler..) in ColumnFamilyStore.apply() and this is also the metric we 
then replicate to Keyspace metrics etc.

This is an odd place for writeLatency. Not to mention it is in a 
hot-path of Memtable-modifications, but it also does not measure the 
real write latency, since it completely ignores the CommitLog latency in 
that same process. Is the intention really to measure 
Memtable-modification latency only or the actual write latencies?

Then the real issue.. this single metric is a cause of huge overhead in 
Memtable processing. There are several metrics / events in the CFS apply 
method, including metric sampler, storageHook reportWrite, 
colUpdateTimeDeltaHistogram and metric.writeLatency. These are not free 
at all when it comes to the processing. I made a small JMH benchmark 
here: https://gist.github.com/burmanm/b5b284bc9f1d410b1d635f6d3dac3ade 
that I'll be referring to.

The most offending of all these metrics is the writeLatency metric. What 
it does is update the latency in codahale's timer, doing a histogram 
update and then going through all the parent metrics also which update 
the keyspace writeLatency and globalWriteLatency. When measuring the 
performance of Memtable.put with parameter of 1 partition (to reduce the 
ConcurrentSkipListMap search speed impact - that's separate issue and 
takes a little bit longer to solve although I've started to prototype 
something..) on my machine I see 1.3M/s performance with the metric and 
when it is disabled the performance climbs to 4M/s. So the overhead for 
this single metric is ~2/3 of total performance. That's insane. My perf 
stats indicate that the CPU is starved as it can't get enough data in.

Removing the replication from TableMetrics to the Keyspace & global 
latencies in the write time (and doing this when metrics are requested 
instead) improves the performance to 2.1M/s on my machine. It's an 
improvement, but it's still huge amount. Even when we pressure the 
ConcurrentSkipListMap with 100 000 partitions in one active Memtable, 
the performance drops by about ~40% due to this metric, so it's never free.

i did not find any discussion replacing the metric processing with 
something faster, so has this been considered before? At least for these 
performance sensitive ones. The other issue is obviously the use of 
System.nanotime() which by itself is very slow (two System.nanotime() 
calls eat another ~1M/s from the performance)

My personal quick fix would be to move writeLatency to Keyspace.apply, 
change write time aggregates to read time processing (metrics are read 
less often than we write data) and maybe even reduce the nanotime -> 
currentTimeMillis (even given it's relative lack of precision). That is 
- if these metrics make any sense at all at CFS level? Maybe these 
should be measured from the network processing time (including all the 
deserializations and such) ? Especially if at some point the smarter 
threading / eventlooping changes go forward (in which case they might 
sleep at some "queue" for a while).

   - Micke


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Reviewer for LWT bug

[Blake Eggleston]

I'll take it


On December 17, 2017 at 3:48:04 PM, kurt greaves (k...@instaclustr.com) wrote:

Need a reviewer for CASSANDRA-14087 
 

Pretty straight forward, we just get an NPE when comparing against a frozen 
collection which is null and we expect a specific collection. Anyone with a 
bit of knowledge around ColumnCondition.java should be able to review. 

Patch is for 3.0 but should apply cleanly to 3.11. Trunk is unaffected. 

Cheers, 
Kurt 

Re: Cassandra pluggable storage engine (update)

[Blake Eggleston]

Hi Dikang,

Cool stuff. 2 questions. Based on your presentation at ngcc, it seems like 
rocks db stores things in byte order. Does this mean that you have code that 
makes each of the existing types byte comparable, or is clustering order 
implementation dependent? Also, I don't see anything in the draft api that 
seems to support splitting the data set into arbitrary categories (ie repaired 
and unrepaired data living in the same token range). Is support for incremental 
repair planned for v1?

Thanks,

Blake


On October 4, 2017 at 1:28:01 PM, Dikang Gu (dikan...@gmail.com) wrote:

Hello C* developers: 

In my previous email 
(https://www.mail-archive.com/dev@cassandra.apache.org/msg11024.html), I 
presented that Instagram was kicking off a project to make C*'s storage engine 
to be pluggable, as other modern databases, like mysql, mongoDB etc, so that 
users will be able to choose most suitable storage engine for different work 
load, or to use different features. In addition to that, a pluggable storage 
engine architecture will improve the modularity of the system, help to increase 
the testability and reliability of Cassandra.

After months of development and testing, we'd like to share the work we have 
done, including the first(draft) version of the C* storage engine API, and the 
first version of the RocksDB based storage engine.

​


For the C* storage engine API, here is the draft version we proposed, 
https://docs.google.com/document/d/1PxYm9oXW2jJtSDiZ-SR9O20jud_0jnA-mW7ttp2dVmk/edit.
 It contains the APIs for read/write requests, streaming, and table management. 
The storage engine related functionalities, like data encoding/decoding format, 
on-disk data read/write, compaction, etc, will be taken care by the storage 
engine implementation.

Each storage engine is a class with each instance of the class is stored in the 
Keyspace instance. So all the column families within a keyspace will share one 
storage engine instance.

Once a storage engine instance is created, Cassandra sever issues commands to 
the engine instance to performance data storage and retrieval tasks such as 
opening a column family, managing column families and streaming.

How to config storage engine for different keyspaces? It's still open for 
discussion. One proposal is that we can add the storage engine option in the 
create keyspace cql command, and potentially we can overwrite the option per C* 
node in its config file.

Under that API, we implemented a new storage engine, based on RocksDB, called 
RocksEngine. In long term, we want to support most of C* existing features in 
RocksEngine, and we want to build it in a progressive manner. For the first 
version of the RocksDBEngine, we support following features:
Most of non-nested data types
Table schema
Point query
Range query
Mutations
Timestamp
TTL
Deletions/Cell tombstones
Streaming
We do not supported following features in first version yet:
Multi-partition query
Nested data types
Counters
Range tombstone
Materialized views
Secondary indexes
SASI
Repair
At this moment, we've implemented the V1 features, and deployed it to our 
shadow cluster. Using shadowing traffic of our production use cases, we saw ~3X 
P99 read latency drop, compared to our C* 2.2 prod clusters. Here are some 
detailed metrics: 
https://docs.google.com/document/d/1DojHPteDPSphO0_N2meZ3zkmqlidRwwe_cJpsXLcp10.

So if you need the features in existing storage engine, please keep using the 
existing storage engine. If you want to have a more predictable and lower read 
latency, also the features supported by RocksEngine are enough for your use 
cases, then RocksEngine could be a fit for you.

The work is 1% finished, and we want to work together with community to make it 
happen. We presented the work in NGCC last week, and also pushed the beta 
version of the pluggable storage engine to Instagram github Cassandra repo, 
rocks_3.0 branch (https://github.com/Instagram/cassandra/tree/rocks_3.0), which 
is based on C* 3.0.12, please feel free to play with it! You can download it 
and follow the instructions 
(https://github.com/Instagram/cassandra/blob/rocks_3.0/StorageEngine.md) to try 
it out in your test environment, your feedback will be very valuable to us.

Thanks
Dikang.

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

The remaining issues are:

* There's no way to determine if a view is out of sync with the base table.
* If you do determine that a view is out of sync, the only way to fix it is to 
drop and rebuild the view.
* There are liveness issues with updates being reflected in the view.

On October 3, 2017 at 9:00:32 AM, Sylvain Lebresne (sylv...@datastax.com) wrote:

On Tue, Oct 3, 2017 at 5:54 PM, Aleksey Yeshchenko  wrote:  
> There are a couple compromise options here:  
>  
> a) Introduce the flag (enalbe_experimental_features, or maybe one per 
> experimental feature), set it to ‘false’ in the yaml, but have the default be 
> ‘true’. So that if you are upgrading from a previous minor to the next 
> without updating the yaml, you notice nothing.  
>  
> b) Introduce the flag in the minor, and set it to ‘true’ in the yaml in 3.0 
> and 3.11, but to ‘false’ in 4.0. So the operators and in general people who 
> know better can still disable it with one flip, but nobody would be affected 
> by it in a minor otherwise.  
>  
> B might be more correct, and I’m okay with it  

Does feel more correct to me as well  

> although I do feel that we are behaving irresponsibly as developers by 
> allowing MV creation by default in their current state  

You're giving little credit to the hard work that people have put into  
getting MV in a usable state. To quote Kurt's email:  

> And finally, back onto the original topic. I'm not convinced that MV's need  
> this treatment now. Zhao and Paulo (and others+reviewers) have made quite a  
> lot of fixes, granted there are still some outstanding bugs but the  
> majority of bad ones have been fixed in 3.11.1 and 3.0.15, the remaining  
> bugs mostly only affect views with a poor data model. Plus we've already  
> required the known broken components require a flag to be turned on.  

-  
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
For additional commands, e-mail: dev-h...@cassandra.apache.org  

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

Yes, I understand what you're saying. The points I'm making about logs still 
apply. It's possible for drivers and object mappers to handle queries and 
schema changes, and have developers rarely open cqlsh. It's also not uncommon 
for schema changes to be done by a different group than the developers writing 
the application.

On October 2, 2017 at 2:21:38 PM, Jeremiah D Jordan (jerem...@datastax.com) 
wrote:

Blake,  
We are not saying to just put something in logs, we are talking about the warn 
actually showing up in cqlsh.  
When you issue a native protocol warn cqlsh will print it out on the console in 
front of you in the results of the query.  
https://issues.apache.org/jira/browse/CASSANDRA-8930 
<https://issues.apache.org/jira/browse/CASSANDRA-8930>  

For example for SASI it would look something like:  


cqlsh:ks> CREATE CUSTOM INDEX ON sasi_table (c) USING 
'org.apache.cassandra.index.sasi.SASIIndex';  

Warnings :  
A SASI index was enabled for ‘ks.sasi_table'. SASI is still experimental, take 
extra caution when using it in production.  

cqlsh:ks>  

-Jeremiah  

> On Oct 2, 2017, at 5:05 PM, Blake Eggleston <beggles...@apple.com> wrote:  
>  
> The message isn't materially different, but it will reach fewer people, 
> later. People typically aren't as attentive to logs as they should be. 
> Developers finding out about new warnings in the logs later than they could 
> have, sometimes even after it's been deployed, is not uncommon. It's happened 
> to me. Requiring a flag will reach everyone trying to use MVs as soon as they 
> start developing against MVs. Logging a warning will reach a subset of users 
> at some point, hopefully. The only downside I can think of for the flag is 
> that it's not as polite.  
>  
> On October 2, 2017 at 1:16:10 PM, Josh McKenzie (jmcken...@apache.org) wrote: 
>  
>  
> "Nobody is talking about removing MVs."  
> Not precisely true for this email thread:  
>  
> "but should there be some point in the  
> future where we consider removing them from the code base unless they have  
> gotten significant improvement as well?"  
>  
> IMO a .yaml change requirement isn't materially different than barfing a  
> warning on someone's screen during the dev process when they use the DDL  
> for MV's. At the end of the day, it's just a question of how forceful you  
> want that messaging to be. If the cqlsh client prints 'THIS FEATURE IS NOT  
> READY' in big bold letters, that's not going to miscommunicate to a user  
> that 'feature X is ready' when it's not.  
>  
> Much like w/SASI, this is something that's in the code-base that for  
> certain use-cases apparently works just fine. Might be worth considering  
> the approach of making boundaries around those use-cases more rigid instead  
> of throwing the baby out with the bathwater.  
>  
> On Mon, Oct 2, 2017 at 3:32 PM, DuyHai Doan <doanduy...@gmail.com> wrote:  
>  
>> Ok so IF there is a flag to enable MV (à-la UDA/UDF in cassandra.yaml) then  
>> I'm fine with it. I initially understood that we wanted to disable it  
>> definitively. Maybe we should then add an explicit error message when MV is  
>> disabled and someone tries to use it, something like:  
>>  
>> "MV has been disabled, to enable it, turn on the flag  in  
>> cassandra.yaml" so users don't spend 3h searching around  
>>  
>>  
>> On Mon, Oct 2, 2017 at 9:07 PM, Jon Haddad <j...@jonhaddad.com> wrote:  
>>  
>>> There’s a big difference between removal of a protocol that every single  
>>> C* user had to use and disabling a feature which is objectively broken  
>> and  
>>> almost nobody is using. Nobody is talking about removing MVs. If you  
>> want  
>>> to use them you can enable them very trivially, but it should be an  
>>> explicit option because they really aren’t ready for general use.  
>>>  
>>> Claiming disabling by default == removal is not helpful to the  
>>> conversation and is very misleading.  
>>>  
>>> Let’s be practical here. The people that are most likely to put MVs in  
>>> production right now are people new to Cassandra that don’t know any  
>>> better. The people that *should* be using MVs are the contributors to  
>> the  
>>> project. People that actually wrote Cassandra code that can do a patch  
>> and  
>>> push it into prod, and get it submitted upstream when they fix something.  
>>> Yes, a lot of this stuff requires production usage to shake out the bugs,  
>>> that’s fine, but we shouldn’t lie to people and say “feature X is ready”  
>>> when it’s not. That’s a great way to get a reputation as “unstabl

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

n  
> > >> the development process.  
> > >>  
> > >> How does emitting a native protocol warning reduce visibility during  
> the  
> > >> development process? If you run CREATE MV and cqlsh then prints out a  
> > >> giant warning statement about how it is an experimental feature I  
> think  
> > >> that is pretty visible during development?  
> > >>  
> > >> I guess I can see just blocking new ones without a flag set, but we  
> need  
> > >> to be careful here. We need to make sure we don’t cause a problem for  
> > >> someone that is using them currently, even with all the edge cases  
> > issues  
> > >> they have now.  
> > >>  
> > >> -Jeremiah  
> > >>  
> > >>  
> > >>> On Oct 2, 2017, at 2:01 PM, Blake Eggleston <beggles...@apple.com>  
> > >> wrote:  
> > >>>  
> > >>> Yeah, I'm not proposing that we disable MVs in existing clusters.  
> > >>>  
> > >>>  
> > >>> On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (  
> > alek...@apple.com)  
> > >> wrote:  
> > >>>  
> > >>> The idea is to check the flag in CreateViewStatement, so creation of  
> > new  
> > >> MVs doesn’t succeed without that flag flipped.  
> > >>>  
> > >>> Obviously, just disabling existing MVs working in a minor would be  
> > silly.  
> > >>>  
> > >>> As for the warning - yes, that should also be emitted.  
> Unconditionally.  
> > >>>  
> > >>> —  
> > >>> AY  
> > >>>  
> > >>> On 2 October 2017 at 18:18:52, Jeremiah D Jordan (  
> > >> jeremiah.jor...@gmail.com) wrote:  
> > >>>  
> > >>> These things are live on clusters right now, and I would not want  
> > >> someone to upgrade their cluster to a new *patch* release and suddenly  
> > >> something that may have been working for them now does not function.  
> > >> Anyway, we need to be careful about how this gets put into practice if  
> > we  
> > >> are going to do it retroactively.  
> > >>  
> > >>  
> > >> -  
> > >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
> > >> For additional commands, e-mail: dev-h...@cassandra.apache.org  
> > >>  
> > >>  
> >  
> >  
> > -  
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
> > For additional commands, e-mail: dev-h...@cassandra.apache.org  
> >  
> >  
>  

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

Yeah, I'm not proposing that we disable MVs in existing clusters.


On October 2, 2017 at 10:58:11 AM, Aleksey Yeshchenko (alek...@apple.com) wrote:

The idea is to check the flag in CreateViewStatement, so creation of new MVs 
doesn’t succeed without that flag flipped.  

Obviously, just disabling existing MVs working in a minor would be silly.  

As for the warning - yes, that should also be emitted. Unconditionally.  

—  
AY  

On 2 October 2017 at 18:18:52, Jeremiah D Jordan (jeremiah.jor...@gmail.com) 
wrote:  

These things are live on clusters right now, and I would not want someone to 
upgrade their cluster to a new *patch* release and suddenly something that may 
have been working for them now does not function. Anyway, we need to be careful 
about how this gets put into practice if we are going to do it retroactively. 

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

Yeah I’m not sure that just emitting a warning is enough. The point is to be 
super explicit that bad things will happen if you use MVs. I would (in a patch 
release) disable MV CREATE statements, and emit warnings for ALTER statements 
and on schema load if they’re not explicitly enabled. Only emitting a warning 
really reduces visibility where we need it: in the development process.

By only emitting warning, we're just protecting users that don't run even 
rudimentary tests before upgrading their clusters. If an operator is going to 
blindly deploy a database update to prod without testing, they’re going to poke 
their eye out on something anyway. Whether it’s an MV flag or something else. 
If we make this change clear in NEWS.txt, and the user@ list, I think that’s 
the best thing to do.


On October 2, 2017 at 10:18:52 AM, Jeremiah D Jordan 
(jeremiah.jor...@gmail.com) wrote:

Hindsight is 20/20. For 8099 this is the reason we cut the 2.2 release before 
8099 got merged.  

But moving forward with where we are now, if we are going to start adding some 
experimental flags to things, then I would definitely put SASI on this list as 
well.  

For both SASI and MV I don’t know that adding a flags in the cassandra.yaml 
which prevents their use is the right way to go. I would propose that we emit 
WARN from the native protocol mechanism when a user does an ALTER/CREATE what 
ever that tries to use an experiment feature, and probably in the system.log as 
well.  So someone who is starting new development using them will get a warning 
showing up in cqlsh “hey the thing you just used is experimental, proceed with 
caution” and also in their logs.  

These things are live on clusters right now, and I would not want someone to 
upgrade their cluster to a new *patch* release and suddenly something that may 
have been working for them now does not function. Anyway, we need to be careful 
about how this gets put into practice if we are going to do it retroactively.  

-Jeremiah  


> On Oct 1, 2017, at 5:36 PM, Josh McKenzie  wrote:  
>  
>>  
>> I think committing 8099, or at the very least, parts of it, behind an  
>> experimental flag would have been the right thing to do.  
>  
> With a major refactor like that, it's a staggering amount of extra work to  
> have a parallel re-write of core components of a storage engine accessible  
> in parallel to the major based on an experimental flag in the same branch.  
> I think the complexity in the code-base of having two such channels in  
> parallel would be an altogether different kind of burden along with making  
> the work take considerably longer. The argument of modularizing a change  
> like that, however, is something I can get behind as a matter of general  
> principle. As we discussed at NGCC, the amount of static state in the C*  
> code-base makes this an aspirational goal rather than a reality all too  
> often, unfortunately.  
>  
> Not looking to get into the discussion of the appropriateness of 8099 and  
> other major refactors like it (nio MessagingService for instance) - but  
> there's a difference between building out new features and shielding the  
> code-base and users from their complexity and reliability and refactoring  
> core components of the code-base to keep it relevant.  
>  
> On Sun, Oct 1, 2017 at 5:01 PM, Dave Brosius  wrote:  
>  
>> triggers  
>>  
>>  
>> On 10/01/2017 11:25 AM, Jeff Jirsa wrote:  
>>  
>>> Historical examples are anything that you wouldn’t bet your job on for  
>>> the first release:  
>>>  
>>> Udf/uda in 2.2  
>>> Incremental repair - would have yanked the flag following 9143  
>>> SASI - probably still experimental  
>>> Counters - all sorts of correctness issues originally, no longer true  
>>> since the rewrite in 2.1  
>>> Vnodes - or at least shuffle  
>>> CDC - is the API going to change or is it good as-is?  
>>> CQL - we’re on v3, what’s that say about v1?  
>>>  
>>> Basically anything where we can’t definitively say “this feature is going  
>>> to work for you, build your product on it” because companies around the  
>>> world are trying to make that determination on their own, and they don’t  
>>> have the same insight that the active committers have.  
>>>  
>>> The transition out we could define as a fixed number of releases or a dev@  
>>> vote, I don’t think you’ll find something that applies to all experimental  
>>> features, so being flexible is probably the best bet there  
>>>  
>>>  
>>>  
>>  
>> -  
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
>> For additional commands, e-mail: dev-h...@cassandra.apache.org  
>>  
>>  


-  
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org  
For additional commands, e-mail: dev-h...@cassandra.apache.org  

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

I think you're presenting a false dichotomy here. Yes there are people who are 
not interested in taking risks with C* and are still running 1.2, there are 
probably a few people who would put trunk in prod if we packaged it up for 
them, but there's a whole spectrum of users in between. Operator competence / 
sophistication has the same sort of spectrum.

I'd expect the amount of feedback on experimental features would be a function 
of the quality of the design / implementation and the amount of user interest. 
If you're not getting feedback on experimental feature, it's probably poorly 
implemented, or no one's interested in it.

I don't think labelling features is going to kill the user <-> developer 
feedback loop. It will probably slow down the pace of feature development a 
bit, but it's been slowing down anyway, and that's a good thing imo.

On October 1, 2017 at 9:14:45 AM, DuyHai Doan (doanduy...@gmail.com) wrote:

So basically we're saying that even with a lot of tests, you're never sure  
to cover all the possible edge cases and the real stamp for "production  
readiness" is only when the "experimental features" have been deployed in  
various clusters with various scenarios/use-cases, just re-phrasing Blake  
here. Totally +1 on the idea.  

Now I can foresee a problem with the "experimental" flag, that is nobody  
(in the community) will use it or even dare to play with it and thus the  
"experimental" features never get a chance to be tested and then we break  
the bug-reports/bug-fixes iterations ...  

How many times have I seen users on the ML asking which version of C* is  
the most fit for production and the answer was always at least 1 major  
version behind the current released major (2.1 was recommended when 3.x was  
released and so one ...) ?  

The fundamental issue here is that a lot of folks in the community do not  
want to take any risk and take a conservative approach for the production,  
which is fine and perfectly understandable. But it means that the implicit  
contract for OSS software, e.g. "you have a software for free in exchange  
you will give feedbacks and bug reports to improve it", is completely  
broken.  

Let's take the example of MV. MV was shipped with 3.0 --> considered not  
stable --> nobody/few people uses MV --> few bug reports --> bugs didn't  
have chance to get fixed --> the problem lasts until now  

About SASI, how many people really played with thoroughly apart from some  
toy examples ? Same causes, same consequences. And we can't even blame its  
design because fundamentally the architecture is pretty solid, just a  
question of usage and feedbacks.  

I suspect that this broken community QA/feedback loop did also explain  
partially the failure of tic/toc releases but it's only my own  
interpretation here.  

So if we don't figure out how to restore the "new feature/community bug  
report" strong feedback loop, we're going to face again the same issues and  
same debate in the future  


On Sun, Oct 1, 2017 at 5:30 PM, Blake Eggleston <beggles...@apple.com>  
wrote:  

> I'm not sure the main issue in the case of MVs is testing. In this case it  
> seems to be that there are some design issues and/or the design was only  
> works in some overly restrictive use cases. That MVs were committed knowing  
> these were issues seems to be the real problem. So in the case of MVs, sure  
> I don't think they should have ever made it to an experimental stage.  
>  
> Thinking of how an experimental flag fits in the with the project going  
> forward though, I disagree that we should avoid adding experimental  
> features. On the contrary, I think leaning towards classifying new features  
> as experimental would be better for users. Especially larger features and  
> changes.  
>  
> Even with well spec'd, well tested, and well designed features, there will  
> always be edge cases that you didn't think of, or you'll have made  
> assumptions about the other parts of C* it relies on that aren't 100%  
> correct. Small problems here can often affect correctness, or result in  
> data loss. So, I think it makes sense to avoid marking them as ready for  
> regular use until they've had time to bake in clusters where there are some  
> expert operators that are sophisticated enough to understand the  
> implications of running them, detect issues, and report bugs.  
>  
> Regarding historical examples, in hindsight I think committing 8099, or at  
> the very least, parts of it, behind an experimental flag would have been  
> the right thing to do. It was a huge change that we're still finding issues  
> with 2 years later.  
>  
> On October 1, 2017 at 6:08:50 AM, DuyHai Doan (doanduy...@gmail.com)  
> wrote:  
>  
> How should we transition one feature from the "experime

Re: Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

I'm not sure the main issue in the case of MVs is testing. In this case it 
seems to be that there are some design issues and/or the design was only works 
in some overly restrictive use cases. That MVs were committed knowing these 
were issues seems to be the real problem. So in the case of MVs, sure I don't 
think they should have ever made it to an experimental stage.

Thinking of how an experimental flag fits in the with the project going forward 
though, I disagree that we should avoid adding experimental features. On the 
contrary, I think leaning towards classifying new features as  experimental 
would be better for users. Especially larger features and changes.

Even with well spec'd, well tested, and well designed features, there will 
always be edge cases that you didn't think of, or you'll have made assumptions 
about the other parts of C* it relies on that aren't 100% correct. Small 
problems here can often affect correctness, or result in data loss. So, I think 
it makes sense to avoid marking them as ready for regular use until they've had 
time to bake in clusters where there are some expert operators that are 
sophisticated enough to understand the implications of running them, detect 
issues, and report bugs.

Regarding historical examples, in hindsight I think committing 8099, or at the 
very least, parts of it, behind an experimental flag would have been the right 
thing to do. It was a huge change that we're still finding issues with 2 years 
later.

On October 1, 2017 at 6:08:50 AM, DuyHai Doan (doanduy...@gmail.com) wrote:

How should we transition one feature from the "experimental" state to  
"production ready" state ? On which criteria ?  



On Sun, Oct 1, 2017 at 12:12 PM, Marcus Eriksson <krum...@gmail.com> wrote:  

> I was just thinking that we should try really hard to avoid adding  
> experimental features - they are experimental due to lack of testing right?  
> There should be a clear path to making the feature non-experimental (or get  
> it removed) and having that path discussed on dev@ might give more  
> visibility to it.  
>  
> I'm also struggling a bit to find good historic examples of "this would  
> have been better off as an experimental feature" - I used to think that it  
> would have been good to commit DTCS with some sort of experimental flag,  
> but that would not have made DTCS any better - it would have been better to  
> do more testing, realise that it does not work and then not commit it at  
> all of course.  
>  
> Does anyone have good examples of features where it would have made sense  
> to commit them behind an experimental flag? SASI might be a good example,  
> but for MVs - if we knew how painful they would be, they really would not  
> have gotten committed at all, right?  
>  
> /Marcus  
>  
> On Sat, Sep 30, 2017 at 7:42 AM, Jeff Jirsa <jji...@gmail.com> wrote:  
>  
> > Reviewers should be able to suggest when experimental is warranted, and  
> > conversation on dev+jira to justify when it’s transitioned from  
> > experimental to stable?  
> >  
> > We should remove the flag as soon as we’re (collectively) confident in a  
> > feature’s behavior - at least correctness, if not performance.  
> >  
> >  
> > > On Sep 29, 2017, at 10:31 PM, Marcus Eriksson <krum...@gmail.com>  
> wrote:  
> > >  
> > > +1 on marking MVs experimental, but should there be some point in the  
> > > future where we consider removing them from the code base unless they  
> > have  
> > > gotten significant improvement as well?  
> > >  
> > > We probably need to enforce some kind of process for adding new  
> > > experimental features in the future - perhaps a mail like this one to  
> > dev@  
> > > motivating why it should be experimental?  
> > >  
> > > /Marcus  
> > >  
> > > On Sat, Sep 30, 2017 at 1:15 AM, Vinay Chella  
> > <vche...@netflix.com.invalid>  
> > > wrote:  
> > >  
> > >> We tried perf testing MVs internally here but did not see good results  
> > with  
> > >> it, hence paused its usage. +1 on tagging certain features which are  
> not  
> > >> PROD ready or not stable enough.  
> > >>  
> > >> Regards,  
> > >> Vinay Chella  
> > >>  
> > >>> On Fri, Sep 29, 2017 at 7:22 PM, Ben Bromhead <b...@instaclustr.com>  
> > wrote:  
> > >>>  
> > >>> I'm a fan of introducing experimental flags in general as well, +1  
> > >>>  
> > >>>  
> > >>>  
> > >>>> On Fri, 29 Sep 2017 at 13:22 Jon Haddad <j...@jonhad

Proposal to retroactively mark materialized views experimental

[Blake Eggleston]

Hi dev@,

I’d like to propose that we retroactively classify materialized views as an 
experimental feature, disable them by default, and require users to enable them 
through a config setting before using.

Materialized views have several issues that make them (effectively) unusable in 
production. Some of the issues aren’t just implementation problems, but 
problems with the design that aren’t easily fixed. It’s unfair of us to make 
features available to users in this state without providing a clear warning 
that bad or unexpected things are likely to happen if they use it.

Obviously, this isn’t great news for users that have already adopted MVs, and I 
don’t have a great answer for that. I think that’s sort of a sunk cost at this 
point. If they have any MV related problems, they’ll have them whether they’re 
marked experimental or not. I would expect this to reduce the number of users 
adopting MVs in the future though, and if they do, it would be opt-in.

Once MVs reach a point where they’re usable in production, we can remove the 
flag. Specifics of how the experimental flag would work can be hammered out in 
a forthcoming JIRA, but I’d imagine it would just prevent users from creating 
new MVs, and maybe log warnings on startup for existing MVs if the flag isn’t 
enabled.

Let me know what you think.

Thanks,

Blake

Re: Proposal: Closing old, unable-to-repro JIRAs

[Blake Eggleston]

+1 to that


On September 14, 2017 at 4:50:54 PM, Jeff Jirsa (jji...@gmail.com) wrote:

There's a number of JIRAs that are old - sometimes very old - that 
represent bugs that either don't exist in modern versions, or don't have 
sufficient information for us to repro, but the reporter has gone away. 

Would anyone be offended if I start tagging these with the label 
'UnableToRepro' or 'Unresponsive' and start a 30 day timer to close them? 
Anyone have a better suggestion?