Re: [VOTE] CEP-25: Trie-indexed SSTable format

[Benjamin Lerer]

+1

Le lun. 19 déc. 2022 à 16:31, Andrés de la Peña  a
écrit :

> +1
>
> On Mon, 19 Dec 2022 at 15:11, Aleksey Yeshchenko 
> wrote:
>
>> +1
>>
>> On 19 Dec 2022, at 13:42, Ekaterina Dimitrova 
>> wrote:
>>
>> +1
>>
>> On Mon, 19 Dec 2022 at 8:30, J. D. Jordan 
>> wrote:
>>
>>> +1 nb
>>>
>>> > On Dec 19, 2022, at 7:07 AM, Brandon Williams 
>>> wrote:
>>> >
>>> > +1
>>> >
>>> > Kind Regards,
>>> > Brandon
>>> >
>>> >> On Mon, Dec 19, 2022 at 6:59 AM Branimir Lambov 
>>> wrote:
>>> >>
>>> >> Hi everyone,
>>> >>
>>> >> I'd like to propose CEP-25 for approval.
>>> >>
>>> >> Proposal:
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-25%3A+Trie-indexed+SSTable+format
>>> >> Discussion:
>>> https://lists.apache.org/thread/3dpdg6dgm3rqxj96cyhn58b50g415dyh
>>> >>
>>> >> The vote will be open for 72 hours.
>>> >> Votes by committers are considered binding.
>>> >> A vote passes if there are at least three binding +1s and no binding
>>> vetoes.
>>> >>
>>> >> Thank you,
>>> >> Branimir
>>>
>>
>>

Re: [DISCUSS] Formation of Apache Cassandra Publicity & Marketing Group

[Benjamin Lerer]

Super happy to see this happening. :-)

Le sam. 21 janv. 2023 à 00:08, Mick Semb Wever  a écrit :

>
> I'll add the both of you, and anyone else that speaks up.
>
> To clarify, being a moderator to the mailing list is only about
> accepting/rejecting posts being sent from recipients that have not (yet)
> subscribed.
> This is usually 95% spam and 5% existing users posting from a
> different account.
>
>
> On Fri, 20 Jan 2023 at 19:24, Molly Monroy  wrote:
>
>> 
>> I am also happy to be a moderator. Melissa and I together can ensure we
>> have a solid level of coverage.
>>
>> On Jan 20, 2023, at 11:03 AM, Melissa Logan 
>> wrote:
>>
>> 
>> I appreciate the open and more structured approach to publicity &
>> marketing so everyone can provide input and for transparency.
>>
>> I'm also happy to be a moderator.
>>
>>
>> On Fri, Jan 20, 2023 at 7:01 AM Patrick McFadin 
>> wrote:
>>
>>> I would be happy to be one of the moderators. Not sure if that's
>>> singular or plural. :D Just need to know how to do it.
>>>
>>> Patrick
>>>
>>> On Fri, Jan 20, 2023 at 1:44 AM Mick Semb Wever  wrote:
>>>
 *To achieve this, we are proposing the formation of a Publicity &
> Marketing Working Group, and we are requesting your participation.*
>


 +1 to the proposal and everything you write Patrick!

 I've submitted the request for the ML (can take 24 hours). Who would
 like to be a moderator for the list?

 Otherwise let's give this a few days for any concerns, questions,
 objections to be raised.


>>
>> --
>> Melissa Logan (she/her)
>> CEO & Founder, Constantia.io
>> LinkedIn
>> 
>>  | Twitter 
>>
>>
>>

Re: [DISCUSSION] Framework for Internal Collection Exposure and Monitoring API Alignment

[Benjamin Lerer]

It seems to me that the question that we need to answer, before Maxim puts
more effort into this work, is: what is the future of JMX?
I agree that we have never been clear about that decision up to now. At
least now we have a reason to clarify that point.

I can start a discussion about that if people agree?

Le dim. 29 janv. 2023 à 10:14, Dinesh Joshi  a écrit :

> I’m also very interested in this area. I quickly skimmed the proposal and
> IIUC it doesn’t call for moving away from JMX. Instead I think it’s making
> it easier to expose metrics over various interfaces. Maxim please correct
> me if I’m wrong in my understanding.
>
> I also second Josh’s point on JMX usage. I know it’s disabled in some
> deployments but it is not common practice in most places.
>
> Dinesh
>
> On Jan 28, 2023, at 4:10 PM, Josh McKenzie  wrote:
>
> 
> First off - thanks so much for putting in this effort Maxim! This is
> excellent work.
>
> Some thoughts on the CEP and responses in thread:
>
> *Considering that JMX is usually not used and disabled in production
> environments for various performance and security reasons, the operator may
> not see the same picture from various of Dropwizard's metrics exporters and
> integrations as Cassandra's JMX metrics provide [1][2].*
>
> I don't think this assertion is true. Cassandra is running in a *lot* of
> places in the world, and JMX has been in this ecosystem for a long time; we
> need data that is basically impossible to get to claim "JMX is usually not
> used in C* environments in prod".
>
> I also wonder about if we should care about JMX?  I know many wish to
> migrate (its going to be a very long time) away from JMX, so do we need a
> wrapper to make JMX and vtables consistent?
>
> If we can move away from a bespoke vtable or JMX based implementation and
> instead have a templatized solution each of these is generated from, that
> to me is the superior option. There's little harm in adding new JMX
> endpoints (or hell, other metrics framework integration?) as a byproduct of
> adding new vtable exposed metrics; we have the same maintenance obligation
> to them as we have to the vtables and if it generates from the same base
> data, we shouldn't have any further maintenance burden due to its presence
> right?
>
> we wish to move away from JMX
>
> I do, and you do, and many people do, but I don't believe *all* people on
> the project do. The last time this came up in slack the conclusion was
> "Josh should go draft a CEP to chart out a path to moving off JMX while
> maintaining backwards-compat w/existing JMX metrics for environments that
> are using them" (so I'm excited to see this CEP pop up before I got to it!
> ;)). Moving to a system that gives us a 0-cost way to keep JMX and vtable
> in sync over time on new metrics seems like a nice compromise for folks
> that have built out JMX-based maintenance infra on top of C*. Plus removing
> the boilerplate toil on vtables. win-win.
>
> If we add a column to the end of the JMX row did we just break users?
>
> I *think* this is arguably true for a vtable / CQL-based solution as well
> from the "you don't know how people are using your API" perspective. Unless
> we have clear guidelines about discretely selecting the columns you want
> from a vtable and trust users to follow them, if people have brittle greedy
> parsers pulling in all data from vtables we could very well break them as
> well by adding a new column right? Could be wrong here; I haven't written
> anything that consumes vtable metric data and maybe the obvious idiom in
> the face of that is robust in the presence of column addition. /shrug
>
> It's certainly more flexible and simpler to write to w/out detonating
> compared to JMX, but it's still an API we'd be revving.
>
> On Sat, Jan 28, 2023, at 4:24 PM, Ekaterina Dimitrova wrote:
>
> Overall I have similar thoughts and questions as David.
>
> I just wanted to add a reminder about this thread from last summer[1]. We
> already have issues with the alignment of JMX and Settings Virtual Table. I
> guess this is how Maxim got inspired to suggest this framework proposal
> which I want to thank him for! (I noticed he assigned CASSANDRA-15254)
>
> Not to open the Pandora box, but to me the most important thing here is to
> come into agreement about the future of JMX and what we will do or not as a
> community. Also, how much time people are able to invest. I guess this will
> influence any directions to be taken here.
>
> [1]
> https://lists.apache.org/thread/8mjcwdyqoobpvw2262bqmskkhs76pp69
>
>
> On Thu, 26 Jan 2023 at 12:41, David Capwell  wrote:
>
> I took a look and I see the result is an interface that looks like the
> vtable interface, that is then used by vtables and JMX?  My first thought
> is why not just use the vtable logic?
>
> I also wonder about if we should care about JMX?  I know many wish to
> migrate (its going to be a very long time) away from JMX, so do we need a
> wrapper to make JMX and vtables consisten

Welcome Patrick McFadin as Cassandra Committer

[Benjamin Lerer]

 The PMC members are pleased to announce that Patrick McFadin has accepted
the invitation to become committer today.

Thanks a lot, Patrick, for everything you have done for this project and
its community through the years.

Congratulations and welcome!

The Apache Cassandra PMC members

Re: Implicitly enabling ALLOW FILTERING on virtual tables

[Benjamin Lerer]

Making ALLOW FILTERING a table option implies giving the right to the
person creating the table the ability to change the way the server will
behave for that table which might not be something that every C* operator
wants. Of course we can allow operators to controle that through the ALLOW
FILTERING guardrail. At that point we would also need to have a default
setting for the entire database.

Le ven. 3 févr. 2023 à 23:44, Miklosovic, Stefan <
stefan.mikloso...@netapp.com> a écrit :

> This is the draft for FILTERING ON|OFF in shell.
>
> I would say this is the most simple solution.
>
> We may still consider table option but what do you think about having it
> simply just set via shell?
>
> https://github.com/apache/cassandra/pull/2141/files
>
> 
> From: Josh McKenzie 
> Sent: Friday, February 3, 2023 23:39
> To: dev
> Subject: Re: Implicitly enabling ALLOW FILTERING on virtual tables
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> they would start to set ALLOW FILTERING here and there in order to not
> think twice about their data model so they can just call it a day.
> Setting this on a per-table basis or having users set this on specific
> queries that hit tables and forgetting they set it are 6 of one and
> half-a-dozen of another.
>
> I like the table property idea personally. That communicates an intent
> about the data model and expectation of the size and usage of data in the
> modeling of the schema that embeds some context and intent there's
> currently no mechanism to communicate.
>
> On Fri, Feb 3, 2023, at 5:00 PM, Miklosovic, Stefan wrote:
> Yes, there would be discrepancy. I do not like that either. If it was only
> about "normal tables vs virtual tables", I could live with that. But the
> fact that there are going to be differences among vtables themselves, that
> starts to be a little bit messy. Then we would need to let operators know
> what tables are always allowed to be filtered on and which do not and that
> just complicates it. Putting that information to comment so it is visible
> in DECSCRIBE is nice idea.
>
> That flag we talk about ... that flag would be used purely internally, it
> would not be in schema to be gossiped.
>
> Also, I am starting to like the suggestion to have something like ALLOW
> FILTERING ON in CQLSH so it would be turned on whole CQL session. That
> leaves tables as they are and it should not be a big deal for operators to
> set. We would have to make sure to add "ALLOW FILTERING" clause to every
> SELECT statement (to virtual tables only?) a user submits. I am not sure if
> this is doable yet though.
>
> 
> From: David Capwell mailto:dcapw...@apple.com>>
> Sent: Friday, February 3, 2023 22:42
> To: dev
> Cc: Maxim Muzafarov
> Subject: Re: Implicitly enabling ALLOW FILTERING on virtual tables
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I don't think the assumption that "virtual tables will always be small and
> always fit in memory" is a safe one.
>
> Agree, there is a repair ticket to have the coordinating node do network
> queries to peers to resolve the table (rather than operator querying
> everything, allow the coordinator node to do it for you)… so this
> assumption may not be true down the line.
>
> I could be open to a table property that says ALLOW FILTERING on by
> default or not… then we can pick and choose vtables (or have vtables
> opt-out)…. I kinda like like the lack of consistency with this approach
> though
>
> On Feb 3, 2023, at 11:24 AM, C. Scott Andreas  > wrote:
>
> There are some ideas that development community members have kicked around
> that may falsify the assumption that "virtual tables are tiny and will fit
> in memory."
>
> One example is CASSANDRA-14629: Abstract Virtual Table for very large
> result sets
> https://issues.apache.org/jira/browse/CASSANDRA-14629
>
> Chris's proposal here is to enable query results from virtual tables to be
> streamed to the client rather than being fully materialized. There are some
> neat possibilities suggested in this ticket, such as debug functionality to
> dump the contents of a raw SSTable via the CQL interface, or the contents
> of the database's internal caches. One could also imagine a feature like
> this providing functionality similar to a foreign data wrapper in other
> databases.
>
> I don't think the assumption that "virtual tables will always be small and
> always fit in memory" is a safe one.
>
> I don't think we should implicitly add "ALLOW FILTERING" to all queries
> against virtual tables because of this, in addition to concern with
> departing from standard CQL semantics for a type of tables deemed special.
>
> – Scott
>
> On Feb 

Re: [VOTE] CEP-21 Transactional Cluster Metadata

[Benjamin Lerer]

+1

Le mar. 7 févr. 2023 à 06:21,  a écrit :

> +1 nb
>
> On Feb 6, 2023, at 9:05 PM, Ariel Weisberg  wrote:
>
> +1
>
> On Mon, Feb 6, 2023, at 11:15 AM, Sam Tunnicliffe wrote:
>
> Hi everyone,
>
> I would like to start a vote on this CEP.
>
> Proposal:
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata
>
> Discussion:
> https://lists.apache.org/thread/h25skwkbdztz9hj2pxtgh39rnjfzckk7
>
> The vote will be open for 72 hours.
> A vote passes if there are at least three binding +1s and no binding
> vetoes.
>
> Thanks,
> Sam
>
>
>

Re: Downgradability

[Benjamin Lerer]

>
> It seems to me the real issue is nobody noticed this was agreed and/or
> forgot and didn’t think about it much.


I have some trouble keeping up with all the discussions those days so I
might have missed the discussion. The only place I remembered where this
subject was mentioned was during the roadmap discussion over a year ago. My
understanding of that discussion was only that there was a wish to build a
solution for that problem. Are you referring to another discussion I
missed?

Le mar. 21 févr. 2023 à 08:41, Jacek Lewandowski <
lewandowski.ja...@gmail.com> a écrit :

> I'd like to mention CASSANDRA-17056 (CEP-17) here as it aims to introduce
> multiple sstable formats support. It allows for providing an implementation
> of SSTableFormat along with SSTableReader and SSTableWriter. That could be
> extended easily to support different implementations for certain version
> ranges, like one impl for ma-nz, other for oa+, etc. without having a
> confusing implementation with a lot of conditional blocks. Old formats in
> such case could be maintained separately from the main code and easily
> switched any time.
>
> thanks
> - - -- --- -  -
> Jacek Lewandowski
>
>
> wt., 21 lut 2023 o 01:46 Yuki Morishita  napisał(a):
>
>> Hi,
>>
>> What I wanted to address in my comment in CASSANDRA-8110(
>> https://issues.apache.org/jira/browse/CASSANDRA-8110?focusedCommentId=17641705&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-17641705)
>> is to focus on better upgrade experience.
>>
>> Upgrading the cluster can be painful for some orgs with mission critical
>> Cassandra cluster, where they cannot tolerate less availability because of
>> the inability to replace the downed node.
>> They also need to plan rolling back to the previous state when something
>> happens along the way.
>> The change I proposed in CASSANDRA-8110 is to achieve the goal of at
>> least enabling SSTable streaming during the upgrade by not upgrading the
>> SSTable version. This can make the cluster to easily rollback to the
>> previous version.
>> Downgrading SSTable is not the primary focus (though Cassandra needs to
>> implement the way to write SSTable in older versions, so it is somewhat
>> related.)
>>
>> I'm preparing the design doc for the change.
>> Also, if I should create a separate ticket from CASSANDRA-8110 for the
>> clarity of the goal of the change, please let me know.
>>
>>
>> On Tue, Feb 21, 2023 at 5:31 AM Benedict  wrote:
>>
>>> FWIW I think 8110 is the right approach, even if it isn’t a panacea. We
>>> will have to eventually also tackle system schema changes (probably not
>>> hard), and may have to think a little carefully about other things, eg with
>>> TTLs the format change is only the contract about what values can be
>>> present, so we have to make sure the data validity checks are consistent
>>> with the format we write. It isn’t as simple as writing an earlier version
>>> in this case (unless we permit truncating the TTL, perhaps)
>>>
>>> On 20 Feb 2023, at 20:24, Benedict  wrote:
>>>
>>>
>>> 
>>> In a self-organising community, things that aren’t self-policed
>>> naturally end up policed in an adhoc manner, and with difficulty. I’m not
>>> sure that’s the same as arbitrary enforcement. It seems to me the real
>>> issue is nobody noticed this was agreed and/or forgot and didn’t think
>>> about it much.
>>>
>>> But, even without any prior agreement, it’s perfectly reasonable to
>>> request that things do not break compatibility if they do not need to, as
>>> part of the normal patch integration process.
>>>
>>> Issues with 3.1->4.0 aren’t particularly relevant as they predate any
>>> agreement to do this. But we can and should address the problem of new
>>> columns in schema tables, as this happens often between versions. I’m not
>>> sure it has in 4.1 though?
>>>
>>> Regarding downgrade versions, surely this should simply be the same as
>>> upgrade versions we support?
>>>
>>>
>>> On 20 Feb 2023, at 20:02, Jeff Jirsa  wrote:
>>>
>>> 
>>> I'm not even convinced even 8110 addresses this - just writing sstables
>>> in old versions won't help if we ever add things like new types or new
>>> types of collections without other control abilities. Claude's other email
>>> in another thread a few hours ago talks about some of these surprises -
>>> "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
>>> added to system/local.  This means that 3.1 system can not read the table
>>> as it has no definition for it.  I tried marking the column for deletion in
>>> the metadata and in the serialization header.  The later got past the
>>> column not found problem, but I suspect that it just means that data
>>> columns after broadcast_port shifted and so incorrectly read." - this is a
>>> harder problem to solve than just versioning sstables and network
>>> protocols.
>>>
>>> Stepping back a bit, we have downgrade ability listed as a goal, but
>>> it's not (as

Re: Regarding CASSANDRA-14227

[Benjamin Lerer]

Hi Manish,
It is unfortunately not an easy bug to fix as it has some significant
impact at different levels of the code base.
The plan is to have the fix as part of 5.0 which should be released later
this year.

Le mar. 21 févr. 2023 à 09:48, manish khandelwal <
manishkhandelwa...@gmail.com> a écrit :

> Hi Team
>
> We are hit by this bug as we approach the deadline as we reach close to
> 2038 as we can see some of our projects breaching the deadline fixed.
>  I can see some progress made in this bug by Berenguer Blasi in last few
> months. I do understand that review timings and development can not be
> predicted but it would be helpful in planning and updating our projects
> with some tentative dates/release when this bug can be targeted.
>
> I would appreciate any indication from community on this.
>
> Regards
> Manish
>

Re: [DISCUSS] Allow UPDATE on settings virtual table to change running configuration

[Benjamin Lerer]

I have seen issues with some updatable parameters which were missing the
volatile keyword.

Le mer. 22 févr. 2023 à 11:36, Aleksey Yeshchenko  a
écrit :

> FWIW most of those volatile fields, if not in fact all of them, should NOT
> be volatile at all. Someone started the trend and most folks have been
> copycatting or doing the same for consistency with the rest of the codebase.
>
> Please definitely don’t rely on that.
>
> On 21 Feb 2023, at 21:06, Maxim Muzafarov  wrote:
>
> 1. Rely on the volatile keyword in front of fields in the Config class;
>
> I would say this is the most confusing option for me because it
> doesn't give us all the guarantees we need, and also:
> - We have no explicit control over what exactly we expose to a user.
> When we modify the JMX API, we're implementing a new method for the
> MBean, which in turn makes this action an explicit exposure;
> - The volatile keyword is not the only way to achieve thread safety,
> and looks strange for the public API design point;
> - A good example is the setEnableDropCompactStorage method, which
> changes the volatile field, but is only visible for testing purposes;
>
>
>

Re: Downgradability

[Benjamin Lerer]

>
> Can somebody explain to me what is so burdensome, that we seem to be
> spending longer debating it than it would take to implement the necessary
> changes?


I believe that we all agree on the outcome. Everybody wants
downgradability. The issue is on the path to get there.

As far as I am concerned, I would like to see a proper solution and as Jeff
suggested the equivalent of the upgrade tests as gatekeepers. Having
everybody trying to enforce it on his own way will only lead to a poor
result in my opinion with some addoc code that might not really guarantee
real downgradability in the end.
We have rushed in the past to get feature outs and pay the price for it. I
simply prefer that we take the time to do things right.

Thanks to Scott and you, downgradability got a much better visibility so no
matter what approach we pick, I am convinced that we will get there.

Le jeu. 23 févr. 2023 à 09:49, Claude Warren, Jr via dev <
dev@cassandra.apache.org> a écrit :

> Broken downgrading can be fixed (I think) by modifying the
> SearializationHeader.toHeader() method where it currently throws an
> UnknownColumnException.  If we can, instead of throwing the exception,
> create a dropped column for the unexpected column then I think the code
> will work.
>
> I realise that to do this in the wild is not possible as it is a change to
> released code, but we could handle it going forward.
>
> On Wed, Feb 22, 2023 at 11:21 PM Henrik Ingo 
> wrote:
>
>> ... ok apparently shift+enter  sends messages now?
>>
>> I was just saying if at least the file format AND system/tables -
>> anything written to disk - can be protected with a switch, then it allows
>> for quick downgrade by shutting down the entire cluster and restarting with
>> the downgraded binary. It's a start.
>>
>> To be able to do that live in a distributed system needs to consider much
>> more: gossip, streaming, drivers, and ultimately all features, because we
>> don't' want an application developer to use a shiny new thing that a) may
>> not be available on all nodes, or b) may disappear if the cluster has to be
>> downgraded later.
>>
>> henrik
>>
>> On Thu, Feb 23, 2023 at 1:14 AM Henrik Ingo 
>> wrote:
>>
>>> Just this once I'm going to be really brief :-)
>>>
>>> Just wanted to share for reference how Mongodb implemented
>>> downgradeability around their 4.4 version:
>>> https://www.mongodb.com/docs/manual/release-notes/6.0-downgrade-sharded-cluster/
>>>
>>> Jeff you're right. Ultimately this is about more than file formats.
>>> However, ideally if at least the
>>>
>>> On Mon, Feb 20, 2023 at 10:02 PM Jeff Jirsa  wrote:
>>>
 I'm not even convinced even 8110 addresses this - just writing sstables
 in old versions won't help if we ever add things like new types or new
 types of collections without other control abilities. Claude's other email
 in another thread a few hours ago talks about some of these surprises -
 "Specifically during the 3.1 -> 4.0 changes a column broadcast_port was
 added to system/local.  This means that 3.1 system can not read the table
 as it has no definition for it.  I tried marking the column for deletion in
 the metadata and in the serialization header.  The later got past the
 column not found problem, but I suspect that it just means that data
 columns after broadcast_port shifted and so incorrectly read." - this is a
 harder problem to solve than just versioning sstables and network
 protocols.

 Stepping back a bit, we have downgrade ability listed as a goal, but
 it's not (as far as I can tell) universally enforced, nor is it clear at
 which point we will be able to concretely say "this release can be
 downgraded to X".   Until we actually define and agree that this is a
 real goal with a concrete version where downgrade-ability becomes real, it
 feels like things are somewhat arbitrarily enforced, which is probably very
 frustrating for people trying to commit work/tickets.

 - Jeff



 On Mon, Feb 20, 2023 at 11:48 AM Dinesh Joshi 
 wrote:

> I’m a big fan of maintaining backward compatibility. Downgradability
> implies that we could potentially roll back an upgrade at any time. While 
> I
> don’t think we need to retain the ability to downgrade in perpetuity it
> would be a good objective to maintain strict backward compatibility and
> therefore downgradability until a certain point. This would imply
> versioning metadata and extending it in such a way that prior version(s)
> could continue functioning. This can certainly be expensive to implement
> and might bloat on-disk storage. However, we could always offer an option
> for the operator to optimize the on-disk structures for the current 
> version
> then we can rewrite them in the latest version. This optimizes the storage
> and opens up new functionality. This means new features that can work with
> o

[DISCUSS] Next release date

[Benjamin Lerer]

Hi,

We forked the 4.0 and 4.1 branches beginning of May. Unfortunately, for 4.1
we were only able to release GA in December which impacted how much time we
could spend focussing on the next release and the progress that we could
do. By consequence, I am wondering if it makes sense for us to branch 5.0
in May or if we should postpone that date.

What is your opinion?

Benjamin

Re: [DISCUSSION] Cassandra + Java 17

[Benjamin Lerer]

Hey Ekaterina,
Thanks for the update and all the work.


> -- we also use setAccessible at numerous places.


It is not necessarily a problem  as long as we do get an issue with the
Modules boundaries and their access. For me it needs to be looked at on a
case by case basis.

- thoughts around the usage/future of Unsafe? History around the choice of
> using it in C* and future plans I might not know of?
>

It was used mainly to get around the fact that Java did not offer other
means to do certain things.
Outside of trying to anticipate some of the restrictions of that API and
make sure that the JDK offers a suitable replacement for us. I am not sure
that there is much that we can do. But I might misunderstand your question.

Le mer. 1 mars 2023 à 21:16, Ekaterina Dimitrova  a
écrit :

> Hi everyone,
> Some updates and questions around JDK 17 below.
> First of all, I wanted to let people know that currently Cassandra trunk
> can be already compiled and run with J8 + J11 + J17. This is the product of
> the realization that the feature branch makes it harder for working on
> JDK17 related tickets due to the involvement of too many moving parts.
> Agreement reached in [1] that new JDK introduction can be done
> incrementally. Scripted UDFs removed, hooks to be added in a follow up
> ticket.
> What does this mean?
> - Currently you can compile and run Cassandra trunk  with JDK 17(further
> to 8+11). You can run unit and java distributed tests already with JDK17
> - CASSANDRA-18106 in progress,  enabling CCM to handle JDK8, 11 and 17
> with trunk and when that is ready we will be able to run also Python tests;
> After that one lands it comes CASSANDRA-18247 ; its goal is to add CircleCI
> config (separate of the one we have for 8+11) for 11+17 which can be used
> from people who work on JDK17 related issues. Patch proposal already in the
> ticket. Final version we will have when we do the switch 8+11 to 11+17,
> things will go through evolution.
>
> What does this mean? Anyone who is interested to test or to help with
> JDK17 effort can easily do it directly from trunk. Jenkins and CircleCI are
> not switched from 8+11 to 11+17 until we are ready. Only test experimental
> additional CircleCI config will be added, temporary to make it easier for
> testing
>
> To remind you - the umbrella ticket for the JDK17 builds is
> CASSANDRA-16895.
> Good outstanding candidate still not assigned - CASSANDRA-18180, if anyone
> has cycles, please, take a look at it. CASSANDRA-18263 might be also of
> interest to someone.
>
> In other news, I added already to the JDK17 jvm options certain
> imports/exports which are needed at this point but as we agreed in the past
> - it will be good to try to eliminate as many of them as we can. Consider
> those experimental in my opinion.
> Some of them though are related to:
> -- some were added already from 11; thoughts?
> *-- *some will be eliminated with some maintenance in progress
> *-- *some are related to
> https://chronicle.software/chronicle-support-java-17. I guess we are
> cornered with those until Chronicle eliminates the need for those.
> (CASSANDRA-18049)
> -- Find a way to get FileDescriptor.fd and sun.nio.ch.FileChannelImpl.fd
> without opening internals (CASSANDRA-17850)
> -- we also use setAccessible at numerous places.
> And I am sure our CI will tell me I am missing something, especially when
> trunk is alive...
>
> A few other questions:
> - thoughts around the usage/future of Unsafe? History around the choice of
> using it in C* and future plans I might not know of?
> - ECJ - It seems the compiler artifacts are moved from here
>  to
> here  and there
> is change of license from EPL1.0 to EPL2.0 too. But if I read correctly
> here  that
> should not affect us. I am dealing with this in CASSANDRA-18190. Please let
> me know if you see any problem with this that I might be missing.
> - Looking at the history of tickets around JMXServerUtils class I guess it
> was accepted that we might have breakages (and we already had
> CASSANDRA-14173) - JmxRegistry extends sun.rmi.registry.RegistryImpl?
>
> Best regards,
> Ekaterina
>
> [1] https://lists.apache.org/thread/c39yrbdszgz9s34vr17wpjdhv6h2oo61
>
>
>

Re: [EXTERNAL] Re: [DISCUSS] Next release date

[Benjamin Lerer]

Sorry, I realized that when I started the discussion I probably did not
frame it enough as I see that it is now going into different directions.
The concerns I am seeing are:
1) A too small amount of time between releases  is inefficient from a
development perspective and from a user perspective. From a development
point of view because we are missing time to deliver some features. From a
user perspective because they cannot follow with the upgrade.
2) Some features are so anticipated (Accord being the one mentioned) that
people would prefer to delay the release to make sure that it is available
as soon as possible.
3) We do not know how long we need to go from the freeze to GA. We hope for
2 months but our last experience was 6 months. So delaying the release
could mean not releasing this year.
4) For people doing marketing it is really hard to promote a product when
you do not know when the release will come and what features might be there.

All those concerns are probably even made worse by the fact that we do not
have a clear visibility on where we are.

Should we clarify that part first by getting an idea of the status of the
different CEPs and other big pieces of work? From there we could agree on
some timeline for the freeze. We could then discuss how to make predictable
the time from freeze to GA.



Le sam. 4 mars 2023 à 18:14, Josh McKenzie  a écrit :

> (for convenience sake, I'm referring to both Major and Minor semver
> releases as "major" in this email)
>
> The big feature from our perspective for 5.0 is ACCORD (CEP-15) and I
> would advocate to delay until this has sufficient quality to be in
> production.
>
> This approach can be pretty unpredictable in this domain; often unforeseen
> things come up in implementation that can give you a long tail on something
> being production ready. For the record - I don't intend to single Accord
> out *at all* on this front, quite the opposite given how much rigor's
> gone into the design and implementation. I'm just thinking from my personal
> experience: everything I've worked on, overseen, or followed closely on
> this codebase always has a few tricks up its sleeve along the way to having
> edge-cases stabilized.
>
> Much like on some other recent topics, I think there's a nuanced middle
> ground where we take things on a case-by-case basis. Some factors that have
> come up in this thread that resonated with me:
>
> For a given potential release date 'X':
> 1. How long has it been since the last release?
> 2. How long do we expect qualification to take from a "freeze" (i.e. no
> new improvement or features, branch) point?
> 3. What body of merged production ready work is available?
> 4. What body of new work do we have high confidence will be ready within Y
> time?
>
> I think it's worth defining a loose "minimum bound and upper bound" on
> release cycles we want to try and stick with barring extenuating
> circumstances. For instance: try not to release sooner than maybe 10 months
> out from a prior major, and try not to release later than 18 months out
> from a prior major. Make exceptions if truly exceptional things land, are
> about to land, or bugs are discovered around those boundaries.
>
> Applying the above framework to what we have in flight, our last release
> date, expectations on CI, etc - targeting an early fall freeze (pending CEP
> status) and mid to late fall or December release "feels right" to me.
>
> With the exception, of course, that if something merges earlier, is
> stable, and we feel is valuable enough to cut a major based on that, we do
> it.
>
> ~Josh
>
> On Fri, Mar 3, 2023, at 7:37 PM, German Eichberger via dev wrote:
>
> Hi,
>
> We shouldn't release just for releases sake. Are there enough new features
> and are they working well enough (quality!).
>
> The big feature from our perspective for 5.0 is ACCORD (CEP-15) and I
> would advocate to delay until this has sufficient quality to be in
> production.
>
> Just because something is released doesn't mean anyone is gonna use it. To
> add some operator perspective: Every time there is a new release we need to
> decide
> 1) are we supporting it
> 2) which other release can we deprecate
>
> and potentially migrate people - which is also a tough sell if there are
> no significant features and/or breaking changes.  So from my perspective
> less frequent releases are better - after all we haven't gotten around to
> support 4.1 🙂
>
> The 5.0 release is also coupled with deprecating  3.11 which is what a
> significant amount of people are using - given 4.1 took longer I am not
> sure how many people are assuming that 5 will be delayed and haven't made
> plans (OpenJDK support for 8 is longer than Java 17 🙂) . So being a bit
> more deliberate with releasing 5.0 and having a longer beta phase are all
> things we should consider.
>
> My 2cts,
> German
>
> *From:* Benedict 
> *Sent:* Wednesday, March 1, 2023 5:59 AM
> *To:* dev@cassandra.apache.org 
> *Subject:* [EXTERNAL] Re: [DISCUSS

Should we cut some new releases?

[Benjamin Lerer]

Hi everybody,

Benedict and Jon recently committed the patch for CASSANDRA-18125
 which fixes some
serious problems at the memtable/flush level. Should we consider cutting
some releases that contain this fix?

Re: Should we cut some new releases?

[Benjamin Lerer]

>
> Not sure about 4.0.9. We released 4.0.8 just few weeks ago. I would do
> 4.1.1 first.


It is true that there have not been many fixes since 4.0.8. Nevertheless
CASSANDRA-18125 <https://issues.apache.org/jira/browse/CASSANDRA-18125> is
a significant issue and 4.0 is certainly more used than 4.1 at this point
so I believe that we should also release a 4.0.9 version.

Le mar. 14 mars 2023 à 17:27, Josh McKenzie  a écrit :

> +1
>
> On Tue, Mar 14, 2023, at 7:50 AM, Aleksey Yeshchenko wrote:
>
> +1
>
> On 14 Mar 2023, at 05:50, Berenguer Blasi 
> wrote:
>
> +1
> On 13/3/23 21:25, Jacek Lewandowski wrote:
>
> +1
>
> pon., 13 mar 2023, 20:36 użytkownik Miklosovic, Stefan <
> stefan.mikloso...@netapp.com> napisał:
>
> Yes, I was waiting for CASSANDRA-18125 to be in.
>
> I can release 4.1.1 to staging tomorrow morning CET if nobody objects that.
>
> Not sure about 4.0.9. We released 4.0.8 just few weeks ago. I would do
> 4.1.1 first.
>
> 
> From: Ekaterina Dimitrova 
> Sent: Monday, March 13, 2023 18:12
> To: dev@cassandra.apache.org
> Subject: Re: Should we cut some new releases?
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> +1
>
> On Mon, 13 Mar 2023 at 12:23, Benjamin Lerer  ble...@apache.org>> wrote:
> Hi everybody,
>
> Benedict and Jon recently committed the patch for CASSANDRA-18125<
> https://issues.apache.org/jira/browse/CASSANDRA-18125> which fixes some
> serious problems at the memtable/flush level. Should we consider cutting
> some releases that contain this fix?
>
>

Re: Welcome our next PMC Chair Josh McKenzie

[Benjamin Lerer]

Congratulations!!!

Le jeu. 23 mars 2023 à 09:31, Berenguer Blasi  a
écrit :

> Congrats Josh!
> On 23/3/23 9:22, Mick Semb Wever wrote:
>
> It is time to pass the baton on, and on behalf of the Apache Cassandra
> Project Management Committee (PMC) I would like to welcome and congratulate
> our next PMC Chair Josh McKenzie (jmckenzie).
>
> Most of you already know Josh, especially through his regular and valuable
> project oversight and status emails, always presenting a balance and
> understanding to the various views and concerns incoming.
>
> Repeating Paulo's words from last year: The chair is an administrative
> position that interfaces with the Apache Software Foundation Board, by
> submitting regular reports about project status and health. Read more about
> the PMC chair role on Apache projects:
> - https://www.apache.org/foundation/how-it-works.html#pmc
> - https://www.apache.org/foundation/how-it-works.html#pmc-chair
> - https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers
>
> The PMC as a whole is the entity that oversees and leads the project and
> any PMC member can be approached as a representative of the committee. A
> list of Apache Cassandra PMC members can be found on:
> https://cassandra.apache.org/_/community.html
>
>

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[Benjamin Lerer]

Hi Doug,

Outside of the changes to the Cassandra Sidecar that are mentioned, what
the CEP proposes is the donation of a library for Spark integration. It
seems to me that this library could be offered as an open source project
outside of the Cassandra project itself. If we accept Spark Bulk Analytic
as part of the Cassandra project it means that the community will commit to
maintain it and ensure that for each Cassandra release it will be fully
compatible. Considering our history with Hadoop integration which has
basically been unmaintained for years, I am not convinced that it is what
we should do.
We only started to expand the scope of the project recently and I would
personally prefer that we do that slowly starting with the drivers that are
critical for C*. Now, it is only my personal opinion and other people might
have a different view on those things.

Le jeu. 23 mars 2023 à 23:29, Miklosovic, Stefan <
stefan.mikloso...@netapp.com> a écrit :

> Hi,
>
> I think this might be a great contribution in the light of removed Hadoop
> integration recently (CASSANDRA-18323) as it will not be in 5.0 anymore. If
> this CEP is adopted and delivered, I can see how it might be a logical
> replacement of that.
>
> Regards
>
> 
> From: Doug Rohrer 
> Sent: Thursday, March 23, 2023 18:33
> To: dev@cassandra.apache.org
> Cc: James Berragan
> Subject: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark
> Bulk Analytics
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> Hi everyone,
>
> Wiki:
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-28%3A+Reading+and+Writing+Cassandra+Data+with+Spark+Bulk+Analytics
>
> We’d like to propose this CEP for adoption by the community.
>
> It is common for teams using Cassandra to find themselves looking for a
> way to interact with large amounts of data for analytics workloads.
> However, Cassandra’s standard APIs aren’t designed for large scale data
> egress/ingest as the native read/write paths weren’t designed for bulk
> analytics.
>
> We’re proposing this CEP for this exact purpose. It enables the
> implementation of custom Spark (or similar) applications that can either
> read or write large amounts of Cassandra data at line rates, by accessing
> the persistent storage of nodes in the cluster via the Cassandra Sidecar.
>
> This CEP proposes new APIs in the Cassandra Sidecar and a companion
> library that allows deep integration into Apache Spark that allows its
> users to bulk import or export data from a running Cassandra cluster with
> minimal to no impact to the read/write traffic.
>
> We will shortly publish a branch with code that will accompany this CEP to
> help readers understand it better.
>
> As a reminder, please keep the discussion here on the dev list vs. in the
> wiki, as we’ve found it easier to manage via email.
>
> Sincerely,
>
> Doug Rohrer & James Berragan
>

Re: [EXTERNAL] Re: [DISCUSS] Next release date

[Benjamin Lerer]

Thanks everybody for the feedback.
It seems to me that considering that Accord (CEP-15) depends on
Transactional Cluster Metadata (CEP-21) and that, according to Sam, CEP-21
could land late July/August, Accord will probably not be fully merged
before late August/September.
That brings us quite late in the year and exposes us to the risk of not
releasing this year. Therefore, I wonder if we should not slightly change
the way we approach the release this year to minimize the risk of not
releasing this year while getting a 5.0 version with all the most
anticipated features.
I would like to propose a partial freeze of 5.0 in June. At which point we
could expect, based on people's feedback, most CEPs and big features,
outside of CEP-21 and CEP-15, to have been committed to trunk. This partial
freeze will be valid for every new feature except CEP-21 and CEP-15. That
approach will allow us to start stabilizing the 5.0 branch before the 2
CEPs are merged and it will also prevent the goal post from moving.
Reducing the time needed to release after the merge.
What do you think?


Le jeu. 16 mars 2023 à 12:38, Mike Adamson  a écrit :

> Sorry, I realised that I hadn't included any completion date for CEP-7. At
> the current time we are looking at completion mid  to end of April.
>
> Mike
>
> On Mon, 13 Mar 2023 at 11:34, Mike Adamson  wrote:
>
>> CEP-7 Storage Attached Index is in review with ~430 files and ~70k LOC.
>> The bulk of the project is in 3 main patches. The first patch (in-memory
>> index and query path) is merged to the feature branch CASSANDRA-16052 and
>> the second patch (on-disk write and literal / string index) is in review.
>>
>> Mike
>>
>> On Thu, 9 Mar 2023 at 09:13, Branimir Lambov  wrote:
>>
>>> CEPs 25 (trie-indexed sstables) and 26 (unified compaction strategy)
>>> should both be ready for review by mid-April.
>>>
>>> Both are around 10k LOC, fairly isolated, and in need of a committer to
>>> review.
>>>
>>> Regards,
>>> Branimir
>>>
>>> On Mon, Mar 6, 2023 at 11:25 AM Benjamin Lerer 
>>> wrote:
>>>
>>>> Sorry, I realized that when I started the discussion I probably did not
>>>> frame it enough as I see that it is now going into different directions.
>>>> The concerns I am seeing are:
>>>> 1) A too small amount of time between releases  is inefficient from a
>>>> development perspective and from a user perspective. From a development
>>>> point of view because we are missing time to deliver some features. From a
>>>> user perspective because they cannot follow with the upgrade.
>>>> 2) Some features are so anticipated (Accord being the one mentioned)
>>>> that people would prefer to delay the release to make sure that it is
>>>> available as soon as possible.
>>>> 3) We do not know how long we need to go from the freeze to GA. We hope
>>>> for 2 months but our last experience was 6 months. So delaying the release
>>>> could mean not releasing this year.
>>>> 4) For people doing marketing it is really hard to promote a product
>>>> when you do not know when the release will come and what features might be
>>>> there.
>>>>
>>>> All those concerns are probably even made worse by the fact that we do
>>>> not have a clear visibility on where we are.
>>>>
>>>> Should we clarify that part first by getting an idea of the status of
>>>> the different CEPs and other big pieces of work? From there we could agree
>>>> on some timeline for the freeze. We could then discuss how to make
>>>> predictable the time from freeze to GA.
>>>>
>>>>
>>>>
>>>> Le sam. 4 mars 2023 à 18:14, Josh McKenzie  a
>>>> écrit :
>>>>
>>>>> (for convenience sake, I'm referring to both Major and Minor semver
>>>>> releases as "major" in this email)
>>>>>
>>>>> The big feature from our perspective for 5.0 is ACCORD (CEP-15) and I
>>>>> would advocate to delay until this has sufficient quality to be in
>>>>> production.
>>>>>
>>>>> This approach can be pretty unpredictable in this domain; often
>>>>> unforeseen things come up in implementation that can give you a long tail
>>>>> on something being production ready. For the record - I don't intend to
>>>>> single Accord out *at all* on this front, quite the opposite given
>>>>> how much rigor's gone into the design and implementation

Re: [EXTERNAL] Re: [DISCUSS] Next release date

[Benjamin Lerer]

>
> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch?


I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and
allowing only CEP-15 and 21 + bug fixes there.
Le ven. 24 mars 2023 à 13:55, Paulo Motta  a
écrit :

> >  I would like to propose a partial freeze of 5.0 in June.
>
> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I agree
> with a branch freeze, but not with trunk freeze.
>
> I might work on small features after June and would be happy to delay
> releasing these on 5.0+, but delaying merge to trunk until 5.0 is released
> could be disruptive to contributors workflows and I would prefer to avoid
> that if possible.
>
> On Fri, Mar 24, 2023 at 6:37 AM Mick Semb Wever  wrote:
>
>>
>> I would like to propose a partial freeze of 5.0 in June.
>>>
>> …
>>>
>> This partial freeze will be valid for every new feature except CEP-21 and
>>> CEP-15.
>>>
>>
>>
>> +1
>>
>> Thanks for summarising the thread this way Benjamin. This addresses my
>> two main concerns: letting the branch/release date slip too much into the
>> unknown, squeezing GA QA efforts, while putting in place exceptional
>> waivers for CEP-21 and CEP-15.
>>
>> I hope that in the future we will be more willing to commit to the
>> release train model: less concerned about "what the next release contains";
>> more comfortable letting big features land where they land. But this is
>> opinion and discussion for another day… possibly looping back to the
>> discussion on preview releases…
>>
>>
>> Do we have yet from anyone a (rough) eta on CEP-15 (post CEP-21) landing
>> in trunk?
>>
>>
>>

Re: [EXTERNAL] [DISCUSS] Next release date

[Benjamin Lerer]

>
> I’m not sure what freezing early for “extra QA time” really buys us?


Tries, SAI, UCS, extended TTLs, Java 17, Dynamic Data Masking... all those
changes potentially introduce their set of issues/flakiness or other
problems. The freeze will allow us to find those early and facilitate the
integration of CEP 21 and 15 in my opinion.

Le ven. 24 mars 2023 à 16:23, Jeremiah D Jordan 
a écrit :

> Given the fundamental change to how cluster operations work coming from
> CEP-21, I’m not sure what freezing early for “extra QA time” really buys
> us?  I wouldn’t trust any multi-node QA done pre commit.
> What “stabilizing” do we expect to be doing during this time?  How much of
> it do we just have to do again after those things merge?  I for one do not
> like to have release branches cut months before their expected release.  It
> just adds extra merge forward and “where should this go”
> questions/overhead.  It could make sense to me to branch branch when CEP-21
> merges and only let in CEP-15 after that.  CEP-15 is mostly “net new stuff”
> and not “changes to existing stuff” from my understanding?  So no QA effort
> wasted if it is done before it merges.
>
> -Jeremiah
>
> On Mar 24, 2023, at 9:38 AM, Josh McKenzie  wrote:
>
> I would like to propose a partial freeze of 5.0 in June
>
> My .02:
> +1 to:
> * partial freeze on an agreed upon date w/agreed upon other things that
> can optionally go in after
> * setting a hard limit on when we ship from that frozen branch regardless
> of whether the features land or not
>
> -1 to:
> * ever feature freezing trunk again. :)
>
> I worry about the labor involved with having very large work like this
> target a frozen branch and then also needing to pull it up to trunk. That
> doesn't sound fun.
>
> If we resurrected the discussion about cutting alpha snapshots from trunk,
> would that change people's perspectives on the weight of this current
> decision? We'd probably also have to re-open pandora's box talking about
> the solidity of our API's on trunk as well if we positioned those alphas as
> being stable enough to start prototyping and/or building future
> applications against.
>
> On Fri, Mar 24, 2023, at 9:59 AM, Brandon Williams wrote:
>
> I am +1 on a 5.0 branch freeze.
>
> Kind Regards,
> Brandon
>
> On Fri, Mar 24, 2023 at 8:54 AM Benjamin Lerer  wrote:
> >>
> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch?
> >
> >
> > I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and
> allowing only CEP-15 and 21 + bug fixes there.
> > Le ven. 24 mars 2023 à 13:55, Paulo Motta  a
> écrit :
> >>
> >> >  I would like to propose a partial freeze of 5.0 in June.
> >>
> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I
> agree with a branch freeze, but not with trunk freeze.
> >>
> >> I might work on small features after June and would be happy to delay
> releasing these on 5.0+, but delaying merge to trunk until 5.0 is released
> could be disruptive to contributors workflows and I would prefer to avoid
> that if possible.
> >>
> >> On Fri, Mar 24, 2023 at 6:37 AM Mick Semb Wever  wrote:
> >>>
> >>>
> >>>> I would like to propose a partial freeze of 5.0 in June.
> >>>>
> >>>> …
> >>>>
> >>>> This partial freeze will be valid for every new feature except CEP-21
> and CEP-15.
> >>>
> >>>
> >>>
> >>> +1
> >>>
> >>> Thanks for summarising the thread this way Benjamin. This addresses my
> two main concerns: letting the branch/release date slip too much into the
> unknown, squeezing GA QA efforts, while putting in place exceptional
> waivers for CEP-21 and CEP-15.
> >>>
> >>> I hope that in the future we will be more willing to commit to the
> release train model: less concerned about "what the next release contains";
> more comfortable letting big features land where they land. But this is
> opinion and discussion for another day… possibly looping back to the
> discussion on preview releases…
> >>>
> >>>
> >>> Do we have yet from anyone a (rough) eta on CEP-15 (post CEP-21)
> landing in trunk?
> >>>
> >>>
>
>
>

Re: [EXTERNAL] [DISCUSS] Next release date

[Benjamin Lerer]

>
> SAI, Accord, UCS, and DDM are all features that can be pretty safely
> disabled/not used, and unlike w/ CEP-21, we *should* be able to de-risk
> those things much more easily before they merge.


That does not mean that the code should not be stabilized before the
release. We had far less features in 4.1 and it took us 6 months to
release. Accepting more features until August will probably result in
extending the time needed to stabilize the release.

I worry about the labor involved with having very large work like this
> target a frozen branch and then also needing to pull it up to trunk. That
> doesn't sound fun.
>
I am not sure I understand the issue of merging the work in a frozen
branch. Should it not facilitate the work if the branch stops changing
heavily? Regarding trunk, if most of us focus on stabilizing the 5.0 branch
or on CEP-15 and 21 I do not think that it will change so much. Am I
missing something?

Le ven. 24 mars 2023 à 19:16, Caleb Rackliffe  a
écrit :

> > I worry about the labor involved with having very large work like this
> target a frozen branch and then also needing to pull it up to trunk. That
> doesn't sound fun.
>
> > I for one do not like to have release branches cut months before their
> expected release.
>
> > CEP-15 is mostly “net new stuff” and not “changes to existing stuff”
> from my understanding?
>
> Yes, yes, and yes. Agree w/ JD here. If we want CEP-21 to make it into
> 5.0, I'd oppose freezing a 5.0 branch for QA until it merges.
>
> SAI, Accord, UCS, and DDM are all features that can be pretty safely
> disabled/not used, and unlike w/ CEP-21, we *should* be able to de-risk
> those things much more easily before they merge.
>
> To try to address Mick's question, assuming we have Accord depending on
> TCM, I'm not sure how closely it can follow TCM in merging. We've been
> talking about this In another thread: "[DISCUSS] cep-15-accord,
> cep-21-tcm, and trunk", but trying to rebase cep-15-accord on cep-21-tcm
> is probably the easiest way to minimize that window...
>
> On Fri, Mar 24, 2023 at 10:41 AM Benjamin Lerer  wrote:
>
>> I’m not sure what freezing early for “extra QA time” really buys us?
>>
>>
>> Tries, SAI, UCS, extended TTLs, Java 17, Dynamic Data Masking... all
>> those changes potentially introduce their set of issues/flakiness or other
>> problems. The freeze will allow us to find those early and facilitate the
>> integration of CEP 21 and 15 in my opinion.
>>
>> Le ven. 24 mars 2023 à 16:23, Jeremiah D Jordan <
>> jeremiah.jor...@gmail.com> a écrit :
>>
>>> Given the fundamental change to how cluster operations work coming from
>>> CEP-21, I’m not sure what freezing early for “extra QA time” really buys
>>> us?  I wouldn’t trust any multi-node QA done pre commit.
>>> What “stabilizing” do we expect to be doing during this time?  How much
>>> of it do we just have to do again after those things merge?  I for one do
>>> not like to have release branches cut months before their expected
>>> release.  It just adds extra merge forward and “where should this go”
>>> questions/overhead.  It could make sense to me to branch branch when CEP-21
>>> merges and only let in CEP-15 after that.  CEP-15 is mostly “net new stuff”
>>> and not “changes to existing stuff” from my understanding?  So no QA effort
>>> wasted if it is done before it merges.
>>>
>>> -Jeremiah
>>>
>>> On Mar 24, 2023, at 9:38 AM, Josh McKenzie  wrote:
>>>
>>> I would like to propose a partial freeze of 5.0 in June
>>>
>>> My .02:
>>> +1 to:
>>> * partial freeze on an agreed upon date w/agreed upon other things that
>>> can optionally go in after
>>> * setting a hard limit on when we ship from that frozen branch
>>> regardless of whether the features land or not
>>>
>>> -1 to:
>>> * ever feature freezing trunk again. :)
>>>
>>> I worry about the labor involved with having very large work like this
>>> target a frozen branch and then also needing to pull it up to trunk. That
>>> doesn't sound fun.
>>>
>>> If we resurrected the discussion about cutting alpha snapshots from
>>> trunk, would that change people's perspectives on the weight of this
>>> current decision? We'd probably also have to re-open pandora's box talking
>>> about the solidity of our API's on trunk as well if we positioned those
>>> alphas as being stable enough to start prototyping and/or building future
>>> applications against.
>>>
&g

Re: [EXTERNAL] [DISCUSS] Next release date

[Benjamin Lerer]

>
> but otherwise I don't recall anything that we could take as an indicator
> that a next release would take a comparable amount of time to 4.1?


Do we have any indicator that proves that it will take less time? We never
managed to do a release in 2 or 3 months so far. Until we have actually
proven that we could do it, I simply prefer assuming that we cannot and
plan for the worst.
We have a lot of significant features that have or will land soon and our
experience suggests that those merges usually bring their set of
instabilities. The goal of the proposal was to make sure that we get rid of
them before TCM and Accord land to allow us to more easily identify the
root causes of problems. Gaining time on the overall stabilization process.
I am fine with people not liking the proposal. Nevertheless, simply hoping
that it will take us 2 months to stabilize the release seems pretty
optimistic to me. Do people have another plan in mind for ensuring a short
stabilization period?


Le lun. 27 mars 2023 à 09:20, Henrik Ingo  a
écrit :

> Not so fast...
>
> There's certainly value in spending that time stabilizing the already done
> features. It's valuable triaging information to say this used to work
> before CEP-21 and only broke after it.
>
> That said, having a very long freeze of trunk, or alternatively having a
> very long lived 5.0 branch that is waiting for Accord and diverging with a
> trunk that is not frozen... are both undesirable options. (A month or two
> could IMO be discussed though.) So I agree with the concern from that point
> of view, I just don't agree that having one batch of big features in
> stabilization period is zero value.
>
>
> henrik
>
>
>
> On Fri, Mar 24, 2023 at 5:23 PM Jeremiah D Jordan <
> jeremiah.jor...@gmail.com> wrote:
>
>> Given the fundamental change to how cluster operations work coming from
>> CEP-21, I’m not sure what freezing early for “extra QA time” really buys
>> us?  I wouldn’t trust any multi-node QA done pre commit.
>> What “stabilizing” do we expect to be doing during this time?  How much
>> of it do we just have to do again after those things merge?  I for one do
>> not like to have release branches cut months before their expected
>> release.  It just adds extra merge forward and “where should this go”
>> questions/overhead.  It could make sense to me to branch branch when CEP-21
>> merges and only let in CEP-15 after that.  CEP-15 is mostly “net new stuff”
>> and not “changes to existing stuff” from my understanding?  So no QA effort
>> wasted if it is done before it merges.
>>
>> -Jeremiah
>>
>> On Mar 24, 2023, at 9:38 AM, Josh McKenzie  wrote:
>>
>> I would like to propose a partial freeze of 5.0 in June
>>
>> My .02:
>> +1 to:
>> * partial freeze on an agreed upon date w/agreed upon other things that
>> can optionally go in after
>> * setting a hard limit on when we ship from that frozen branch regardless
>> of whether the features land or not
>>
>> -1 to:
>> * ever feature freezing trunk again. :)
>>
>> I worry about the labor involved with having very large work like this
>> target a frozen branch and then also needing to pull it up to trunk. That
>> doesn't sound fun.
>>
>> If we resurrected the discussion about cutting alpha snapshots from
>> trunk, would that change people's perspectives on the weight of this
>> current decision? We'd probably also have to re-open pandora's box talking
>> about the solidity of our API's on trunk as well if we positioned those
>> alphas as being stable enough to start prototyping and/or building future
>> applications against.
>>
>> On Fri, Mar 24, 2023, at 9:59 AM, Brandon Williams wrote:
>>
>> I am +1 on a 5.0 branch freeze.
>>
>> Kind Regards,
>> Brandon
>>
>> On Fri, Mar 24, 2023 at 8:54 AM Benjamin Lerer  wrote:
>> >>
>> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch?
>> >
>> >
>> > I was thinking of a cassandra-5.0 branch freeze. So branching 5.0 and
>> allowing only CEP-15 and 21 + bug fixes there.
>> > Le ven. 24 mars 2023 à 13:55, Paulo Motta  a
>> écrit :
>> >>
>> >> >  I would like to propose a partial freeze of 5.0 in June.
>> >>
>> >> Would that be a trunk freeze, or freeze of a cassandra-5.0 branch? I
>> agree with a branch freeze, but not with trunk freeze.
>> >>
>> >> I might work on small features after June and would be happy to delay
>> releasing these on 5.0+, but delaying merge to trunk until 5.0 is released
>&g

Re: [VOTE] CEP-26: Unified Compaction Strategy

[Benjamin Lerer]

+1

Le mar. 4 avr. 2023 à 17:17, Andrés de la Peña  a
écrit :

> +1
>
> On Tue, 4 Apr 2023 at 15:09, Jeremy Hanna 
> wrote:
>
>> +1 nb, will be great to have this in the codebase - it will make nearly
>> every table's compaction work more efficiently.  The only possible
>> exception is tables that are well suited for TWCS.
>>
>> On Apr 4, 2023, at 8:00 AM, Berenguer Blasi 
>> wrote:
>>
>> +1
>> On 4/4/23 14:36, J. D. Jordan wrote:
>>
>> +1
>>
>> On Apr 4, 2023, at 7:29 AM, Brandon Williams 
>>  wrote:
>>
>> 
>> +1
>>
>> On Tue, Apr 4, 2023, 7:24 AM Branimir Lambov  wrote:
>>
>>> Hi everyone,
>>>
>>> I would like to put CEP-26 to a vote.
>>>
>>> Proposal:
>>>
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-26%3A+Unified+Compaction+Strategy
>>>
>>> JIRA and draft implementation:
>>> https://issues.apache.org/jira/browse/CASSANDRA-18397
>>>
>>> Up-to-date documentation:
>>>
>>> https://github.com/blambov/cassandra/blob/CASSANDRA-18397/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md
>>>
>>> Discussion:
>>> https://lists.apache.org/thread/8xf5245tclf1mb18055px47b982rdg4b
>>>
>>> The vote will be open for 72 hours.
>>> A vote passes if there are at least three binding +1s and no binding
>>> vetoes.
>>>
>>> Thanks,
>>> Branimir
>>>
>>
>>

Re: [VOTE] Release Apache Cassandra 4.0.9 - SECOND ATTEMPT

[Benjamin Lerer]

+1

Le jeu. 13 avr. 2023 à 08:56, Tommy Stendahl via dev <
dev@cassandra.apache.org> a écrit :

> +1 (nb)
>
> -Original Message-
> *From*: Brandon Williams  >
> *Reply-To*: dev@cassandra.apache.org
> *To*: dev@cassandra.apache.org
> *Subject*: Re: [VOTE] Release Apache Cassandra 4.0.9 - SECOND ATTEMPT
> *Date*: Tue, 11 Apr 2023 05:30:59 -0500
>
> +1
>
>
>
> On Tue, Apr 11, 2023 at 2:54 AM Miklosovic, Stefan
>
> <
>
> stefan.mikloso...@netapp.com
>
> > wrote:
>
>
> Lets just vote on that straight away. Nothing significant has changed except 
> zstd-jni update to 1.5.5. If all goes well it would be nice to have the vote 
> resolved by this Friday's noon UTC.
>
>
> Proposing the test build of Cassandra 4.0.9 for release.
>
>
> sha1: e9f8f2efa2ba75f223f31ca6801aff3fe2964745
>
> Git:
>
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0.9-tentative
>
>
> Maven Artifacts:
>
> https://repository.apache.org/content/repositories/orgapachecassandra-1286/org/apache/cassandra/cassandra-all/4.0.9/
>
>
>
> The Source and Build Artifacts, and the Debian and RPM packages and 
> repositories, are available here:
>
> https://dist.apache.org/repos/dist/dev/cassandra/4.0.9/
>
>
>
> The vote will be open for 72 hours (longer if needed). Everyone who has 
> tested the build is invited to vote. Votes by PMC members are considered 
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
>
> [1]: CHANGES.txt:
>
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0.9-tentative
>
>
> [2]: NEWS.txt:
>
> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0.9-tentative
>
>

Re: [DISCUSS] CEP-8 Drivers Donation - take 2

[Benjamin Lerer]

The idea was to have a single driver sub-project. Even if the code bases
are different we believe that it is important to keep the drivers together
to retain cohesive API semantics and make sure they have similar
functionality and feature support.
In this scenario we would need only 3 PMC members for the governance. I am
willing to be one of them.

For the committers, my understanding, based on subproject governance
procedures,
 was that
they should be proposed directly to the PMC members.

Is the vote for the CEP to be for all drivers, but we will introduce each
> driver one by one?  What determines when we are comfortable with one driver
> subproject and can move on to accepting the next ?
>

The goal of the CEP is simply to ensure that the community is in favor of
the donation. Nothing more.
The plan is to introduce the drivers, one by one. Each driver donation will
need to be accepted first by the PMC members, as it is the case for any
donation. Therefore the PMC should have full control on the pace at which
new drivers are accepted.


Le mar. 30 mai 2023 à 12:22, Josh McKenzie  a écrit :

> Is the vote for the CEP to be for all drivers, but we will introduce each
> driver one by one?  What determines when we are comfortable with one driver
> subproject and can move on to accepting the next ?
>
> Curious to hear on this as well. There's 2 implications from the CEP as
> written:
>
> 1. The Java and Python drivers hold special importance due to their
> language proximity and/or project's dependence upon them (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation#CEP8:DatastaxDriversDonation-Scope
> )
> 2. Datastax is explicitly offering all 7 drivers for donation (
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation#CEP8:DatastaxDriversDonation-Goals
> )
>
> This is the most complex contribution via CEP thus far from a governance
> perspective; I suggest we chart a bespoke path to navigate this. Having a
> top level indication of "the CEP is approved" logically separate from a
> per-language indication of "the project is ready to absorb this language
> driver now" makes sense to me. This could look like:
>
> * Vote on the CEP itself
> * Per language (processing one at a time):
> * identify 3 PMC members willing to take on the governance role for
> the language driver
> * Identify 2 contributors who are active on a given driver and
> stepping forward for a committer role on the driver
> * Vote on inclusion of that language driver in the project + commit
> bits
> * Integrate that driver into the project ecosystem (build, ci, docs,
> etc)
>
> Not sure how else we could handle committers / contributors / PMC members
> other than on a per-driver basis.
>
> On Tue, May 30, 2023, at 5:36 AM, Mick Semb Wever wrote:
>
>
> Thank you so much Jeremy and Greg (+others) for all the hard work on this.
>
>
>
> At this point, we'd like to propose CEP-8 for consideration, starting the
> process to accept the DataStax Java driver as an official ASF project.
>
>
>
> Is the vote for the CEP to be for all drivers, but we will introduce each
> driver one by one?  What determines when we are comfortable with one driver
> subproject and can move on to accepting the next ?
>
> Are there key committers and contributors on each driver that want to be
> involved?  Should they be listed before the vote?
> We also need three PMC for the new subproject.  Are we to assign these
> before the vote?
>
>
>
>

Re: [DISCUSS] Limiting query results by size (CASSANDRA-11745)

[Benjamin Lerer]

Thanks Jacek for raising that discussion.

I do not have in mind a scenario where it could be useful to specify a
LIMIT in bytes. The LIMIT clause is usually used when you know how many
rows you wish to display or use. Unless somebody has a useful scenario in
mind I do not think that there is a need for that feature.

Paging in bytes makes sense to me as the paging mechanism is transparent
for the user in most drivers. It is simply a way to optimize your memory
usage from end to end.

I do not like the approach of using both of them simultaneously because if
you request a page with a certain amount of rows and do not get it then is
is really confusing and can be a problem for some usecases. We have users
keeping their session open and the page information to display page of data.

Le lun. 12 juin 2023 à 09:08, Jacek Lewandowski 
a écrit :

> Hi,
>
> I was working on limiting query results by their size expressed in bytes,
> and some questions arose that I'd like to bring to the mailing list.
>
> The semantics of queries (without aggregation) - data limits are applied
> on the raw data returned from replicas - while it works fine for the row
> number limits as the number of rows is not likely to change after
> post-processing, it is not that accurate for size based limits as the cell
> sizes may be different after post-processing (for example due to applying
> some transformation function, projection, or whatever).
>
> We can truncate the results after post-processing to stay within the
> user-provided limit in bytes, but if the result is smaller than the limit -
> we will not fetch more. In that case, the meaning of "limit" being an
> actual limit is valid though it would be misleading for the page size
> because we will not fetch the maximum amount of data that does not exceed
> the page size.
>
> Such a problem is much more visible for "group by" queries with
> aggregation. The paging and limiting mechanism is applied to the rows
> rather than groups, as it has no information about how much memory a single
> group uses. For now, I've approximated a group size as the size of the
> largest participating row.
>
> The problem concerns the allowed interpretation of the size limit
> expressed in bytes. Whether we want to use this mechanism to let the users
> precisely control the size of the resultset, or we instead want to use this
> mechanism to limit the amount of memory used internally for the data and
> prevent problems (assuming restricting size and rows number can be used
> simultaneously in a way that we stop when we reach any of the specified
> limits).
>
> https://issues.apache.org/jira/browse/CASSANDRA-11745
>
> thanks,
> - - -- --- -  -
> Jacek Lewandowski
>

Re: [DISCUSS] Limiting query results by size (CASSANDRA-11745)

[Benjamin Lerer]

>
> If you have rows that vary significantly in their size, your latencies
> could end up being pretty unpredictable using a LIMIT BY . Being
> able to specify a limit by bytes at the driver / API level would allow app
> devs to get more deterministic results out of their interaction w/the DB if
> they're looking to respond back to a client within a certain time frame and
> / or determine next steps in the app (continue paging, stop, etc) based on
> how long it took to get results back.


Are you talking about the page size or the LIMIT. Once the LIMIT is reached
there is no "continue paging". LIMIT is also at the CQL level not at the
driver level.
I can totally understand the need for a page size in bytes not for a LIMIT.

Le lun. 12 juin 2023 à 16:25, Josh McKenzie  a écrit :

> I do not have in mind a scenario where it could be useful to specify a
> LIMIT in bytes. The LIMIT clause is usually used when you know how many
> rows you wish to display or use. Unless somebody has a useful scenario in
> mind I do not think that there is a need for that feature.
>
> If you have rows that vary significantly in their size, your latencies
> could end up being pretty unpredictable using a LIMIT BY . Being
> able to specify a limit by bytes at the driver / API level would allow app
> devs to get more deterministic results out of their interaction w/the DB if
> they're looking to respond back to a client within a certain time frame and
> / or determine next steps in the app (continue paging, stop, etc) based on
> how long it took to get results back.
>
> I'm seeing similar tradeoffs working on gracefully paging over tombstones;
> there's a strong desire to be able to have more confidence in the statement
> "If I ask the server for a page of data, I'll very likely get it back
> within time X".
>
> There's an argument that it's a data modeling problem and apps should
> model differently to have more consistent row sizes and/or tombstone
> counts; I'm sympathetic to that but the more we can loosen those
> constraints on users the better their experience in my opinion.
>
> On Mon, Jun 12, 2023, at 5:39 AM, Jacek Lewandowski wrote:
>
> Yes, LIMIT BY  provided by the user in CQL does not make much sense
> to me either
>
>
> pon., 12 cze 2023 o 11:20 Benedict  napisał(a):
>
>
> I agree that this is more suitable as a paging option, and not as a CQL
> LIMIT option.
>
> If it were to be a CQL LIMIT option though, then it should be accurate
> regarding result set IMO; there shouldn’t be any further results that could
> have been returned within the LIMIT.
>
>
> On 12 Jun 2023, at 10:16, Benjamin Lerer  wrote:
>
> 
> Thanks Jacek for raising that discussion.
>
> I do not have in mind a scenario where it could be useful to specify a
> LIMIT in bytes. The LIMIT clause is usually used when you know how many
> rows you wish to display or use. Unless somebody has a useful scenario in
> mind I do not think that there is a need for that feature.
>
> Paging in bytes makes sense to me as the paging mechanism is transparent
> for the user in most drivers. It is simply a way to optimize your memory
> usage from end to end.
>
> I do not like the approach of using both of them simultaneously because if
> you request a page with a certain amount of rows and do not get it then is
> is really confusing and can be a problem for some usecases. We have users
> keeping their session open and the page information to display page of data.
>
> Le lun. 12 juin 2023 à 09:08, Jacek Lewandowski <
> lewandowski.ja...@gmail.com> a écrit :
>
> Hi,
>
> I was working on limiting query results by their size expressed in bytes,
> and some questions arose that I'd like to bring to the mailing list.
>
> The semantics of queries (without aggregation) - data limits are applied
> on the raw data returned from replicas - while it works fine for the row
> number limits as the number of rows is not likely to change after
> post-processing, it is not that accurate for size based limits as the cell
> sizes may be different after post-processing (for example due to applying
> some transformation function, projection, or whatever).
>
> We can truncate the results after post-processing to stay within the
> user-provided limit in bytes, but if the result is smaller than the limit -
> we will not fetch more. In that case, the meaning of "limit" being an
> actual limit is valid though it would be misleading for the page size
> because we will not fetch the maximum amount of data that does not exceed
> the page size.
>
> Such a problem is much more visible for "group by" queries with
> aggregation. The paging and limi

Re: [DISCUSS] Limiting query results by size (CASSANDRA-11745)

[Benjamin Lerer]

>
> So my other question - for aggregation with the "group by" clause, we
> return an aggregated row which is computed from a group of rows - with my
> current implementation, it is approximated by counting the size of the
> largest row in that group - I think it is the safest and simplest
> approximation - wdyt?


I feel that there are something that was not discussed here. The storage
engine can return some rows that are much larger than the actual row
returned to the user depending on the projections being used. Therefore
there will only be a reliable matching between the size of the page loaded
internally and the size of the page returned to the user when the full row
is queried without transformation. For all the other case the difference
can be really significant. For a group by queries doing a count(*), the
approach suggested will return a page size that is totally off with what
was requested.

Le mar. 13 juin 2023 à 07:00, Jacek Lewandowski 
a écrit :

> Josh, that answers my question exactly; thank you.
>
> I will not implement limiting the result set in CQL (that is, by LIMIT
> clause) and stay with just paging. Whether the page size is defined in
> bytes or rows can be determined by a flag - there are many unused bits for
> that.
>
> So my other question - for aggregation with the "group by" clause, we
> return an aggregated row which is computed from a group of rows - with my
> current implementation, it is approximated by counting the size of the
> largest row in that group - I think it is the safest and simplest
> approximation - wdyt?
>
>
> pon., 12 cze 2023 o 22:55 Josh McKenzie  napisał(a):
>
>> As long as it is valid in the paging protocol to return a short page, but
>> still say “there are more pages”, I think that is fine to do that.
>>
>> Thankfully the v3-v5 spec all make it clear that clients need to respect
>> what the server has to say about there being more pages:
>> https://github.com/apache/cassandra/blob/trunk/doc/native_protocol_v5.spec#L1247-L1253
>>
>>   - Clients should not rely on the actual size of the result set returned
>> to
>> decide if there are more results to fetch or not. Instead, they
>> should always
>> check the Has_more_pages flag (unless they did not enable paging for
>> the query
>> obviously). Clients should also not assert that no result will have
>> more than
>>  results. While the current implementation always
>> respects
>> the exact value of , we reserve the right to return
>> slightly smaller or bigger pages in the future for performance
>> reasons.
>>
>>
>> On Mon, Jun 12, 2023, at 3:19 PM, Jeremiah Jordan wrote:
>>
>> As long as it is valid in the paging protocol to return a short page, but
>> still say “there are more pages”, I think that is fine to do that.  For an
>> actual LIMIT that is part of the user query, I think the server must always
>> have returned all data that fits into the LIMIT when all pages have been
>> returned.
>>
>> -Jeremiah
>>
>> On Jun 12, 2023 at 12:56:14 PM, Josh McKenzie 
>> wrote:
>>
>>
>> Yeah, my bad. I have paging on the brain. Seriously.
>>
>> I can't think of a use-case in which a LIMIT based on # bytes makes sense
>> from a user perspective.
>>
>> On Mon, Jun 12, 2023, at 1:35 PM, Jeff Jirsa wrote:
>>
>>
>>
>> On Mon, Jun 12, 2023 at 9:50 AM Benjamin Lerer  wrote:
>>
>> If you have rows that vary significantly in their size, your latencies
>> could end up being pretty unpredictable using a LIMIT BY . Being
>> able to specify a limit by bytes at the driver / API level would allow app
>> devs to get more deterministic results out of their interaction w/the DB if
>> they're looking to respond back to a client within a certain time frame and
>> / or determine next steps in the app (continue paging, stop, etc) based on
>> how long it took to get results back.
>>
>>
>> Are you talking about the page size or the LIMIT. Once the LIMIT is
>> reached there is no "continue paging". LIMIT is also at the CQL level not
>> at the driver level.
>> I can totally understand the need for a page size in bytes not for a
>> LIMIT.
>>
>>
>> Would only ever EXPECT to see a page size in bytes, never a LIMIT
>> specifying bytes.
>>
>> I know the C-11745 ticket says LIMIT, too, but that feels very odd to me.
>>
>>
>>
>>

Re: [VOTE] CEP-8 Datastax Drivers Donation

[Benjamin Lerer]

+1

Le mar. 13 juin 2023 à 16:46, Ekaterina Dimitrova  a
écrit :

> +1
>
> On Tue, 13 Jun 2023 at 10:33, Jeff Jirsa  wrote:
>
>> +1
>>
>>
>> On Tue, Jun 13, 2023 at 7:15 AM Jeremy Hanna 
>> wrote:
>>
>>> Calling for a vote on CEP-8 [1].
>>>
>>> To clarify the intent, as Benjamin said in the discussion thread [2],
>>> the goal of this vote is simply to ensure that the community is in
>>> favor of the donation. Nothing more.
>>> The plan is to introduce the drivers, one by one. Each driver donation
>>> will need to be accepted first by the PMC members, as it is the case for
>>> any donation. Therefore the PMC should have full control on the pace at
>>> which new drivers are accepted.
>>>
>>> If this vote passes, we can start this process for the Java driver under
>>> the direction of the PMC.
>>>
>>> Jeremy
>>>
>>> 1.
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
>>> 2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp
>>>
>>

[DISCUSS] Slab allocation and memory measurements

[Benjamin Lerer]

Hi,

While working on integrating the new version of Jamm to Cassandra, I
realized that our way to create slabs and how we measure their memory
consumption may not be optimal.

For the sake of simplicity I will only talk about read-write heap buffers
here. Considering that the same principles can be applied to other types of
buffers.

Looking at
*org.apache.cassandra.utils.memory.SlabAllocator.Region::allocate* what we
do is:
data.duplicate().position((newOffset)).limit(newOffset + size)

This results in a ByteBuffer with the same capacity as the original one
with a position and a limit that can be moved outside the new limit that we
defined.
>From a measurement perspective if we want to avoid taking into account the
shared array structure for each buffer, we need to be able to determine if
your buffer can be considered as a slab or not. The current approach is to
consider as a slab anything where the underlying array size is greater than
the number of remaining bytes. This approach seems fragile to me.

If we were using the slice method to create our slab:
data.duplicate().position((newOffset)).limit(newOffset + size).slice()
The slab ByteBuffer would have a capacity that represent the real size of
the slab and will prevent us to change the position or limit to an
incorrect value. It will allow us to reliably identify a slab buffer as its
capacity will always be smaller than the underlying array.

For DirectBuffer using slice after a duplicate is not a good idea before
Java 12 due to a Java bug (https://bugs.openjdk.org/browse/JDK-8208362)
which would result in using 64 extra bytes by direct slab buffer.
Nevertheless, I was wondering if there were other reasons for not using
slice when allocating slabs and if we should not consider using them for
heap buffer for the moment and for all buffers once we do not support only
Java 17 and 21.

Re: [VOTE] Release Apache Cassandra 5.0-alpha1 (take3)

[Benjamin Lerer]

+1

Le jeu. 7 sept. 2023 à 13:53, Jacek Lewandowski 
a écrit :

> Mick, is the documentation / website ok?
>
> If so, +1
>
> Best Regards,
> - - -- --- -  -
> Jacek Lewandowski
>
>
> czw., 7 wrz 2023 o 12:58 Brandon Williams  napisał(a):
>
>> +1
>>
>> Kind Regards,
>> Brandon
>>
>> On Mon, Sep 4, 2023 at 3:26 PM Mick Semb Wever  wrote:
>> >
>> >
>> > Proposing the test build of Cassandra 5.0-alpha1 for release.
>> >
>> > DISCLAIMER, this alpha release does not contain the expected 5.0
>> > features: Vector Search (CEP-30), Transactional Cluster Metadata
>> > (CEP-21) and Accord Transactions (CEP-15).  These features will land
>> > in a later alpha release.
>> >
>> > Please also note that this is an alpha release and what that means,
>> further info at
>> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>> >
>> > sha1: bc5e3741d475e2e99fd7a10450681fd708431a89
>> > Git: https://github.com/apache/cassandra/tree/5.0-alpha1-tentative
>> > Maven Artifacts:
>> https://repository.apache.org/content/repositories/orgapachecassandra-1316/org/apache/cassandra/cassandra-all/5.0-alpha1/
>> >
>> > The Source and Build Artifacts, and the Debian and RPM packages and
>> repositories, are available here:
>> https://dist.apache.org/repos/dist/dev/cassandra/5.0-alpha1/
>> >
>> > The vote will be open for 72 hours (longer if needed). Everyone who has
>> tested the build is invited to vote. Votes by PMC members are considered
>> binding. A vote passes if there are at least three binding +1s and no -1's.
>> >
>> > [1]: CHANGES.txt:
>> https://github.com/apache/cassandra/blob/5.0-alpha1-tentative/CHANGES.txt
>> > [2]: NEWS.txt:
>> https://github.com/apache/cassandra/blob/5.0-alpha1-tentative/NEWS.txt
>> >
>>
>

[DISCUSS] Vector type and empty value

[Benjamin Lerer]

Hi everybody,

I noticed that the new Vector type accepts empty ByteBuffer values as an
input representing null.
When we introduced TINYINT and SMALLINT (CASSANDRA-895) we started making
types non -emptiable. This approach makes more sense to me as having to
deal with empty value is error prone in my opinion.
I also think that it would be good to standardize on one approach to avoid
confusion.

Should we make the Vector type non-emptiable and stick to it for the new
types?

I like to hear your opinion.

Re: [VOTE] Accept java-driver

[Benjamin Lerer]

+1

Le jeu. 5 oct. 2023 à 13:46, Aleksey Yeshchenko  a
écrit :

> +1
>
> On 5 Oct 2023, at 10:35, Benedict  wrote:
>
> Surely it needs to be shared with the foundation and the PMC so we can
> verify? Or at least have ASF legal confirm they have received and are
> satisfied with the tarball? It certainly can’t be kept private to DS,
> AFAICT.
>
> Of course it shouldn’t be shared publicly but not sure how PMC can fulfil
> its verification function here without it.
>
> On 5 Oct 2023, at 10:23, Mick Semb Wever  wrote:
>
> 
>
>.
>
> On Tue, 3 Oct 2023 at 13:25, Josh McKenzie  wrote:
>
>> I see now this will likely be instead apache/cassandra-java-driver
>>
>> I was wondering about that. apache/java-driver seemed pretty broad. :)
>>
>> From the linked page:
>> Check that all active committers have a signed CLA on record. TODO –
>> attach list
>> I've been part of these discussions and work so am familiar with the
>> status of it (as well as guidance and clearance from the foundation re:
>> folks we couldn't reach) - but might be worthwhile to link to the sheet or
>> perhaps instead provide a summary of the 49 java contributors, their CLA
>> signing status, attempts to reach out, etc for other PMC members that
>> weren't actively involved back when we were working through it.
>>
>
>
> We have a spreadsheet with this information, and the tarball of all the
> signed CLAs.
> The tarball we should keep private to DS, but know that we have it for
> governance's sake.
>
> I've attached the spreadsheet to the CEP confluence page.
>
>
>

Re: [DISCUSS] putting versions into Deprecated annotations

[Benjamin Lerer]

I am a bit confused by the starting point of this discussion: "When we
deprecate APIs / methods"
What are we exactly calling APIs/methods? It is really unclear to me what
we are talking about here.

Le jeu. 12 oct. 2023 à 02:38, Francisco Guerrero  a
écrit :

>
>
> On 2023/10/11 16:59:35 Maxim Muzafarov wrote:
> > Francisco,
> >
> > I agree with your vision of the deprecation comments and actually, I
> > think we should recommend doing it that way for the cases where it is
> > applicable on our code-style page, but when things get to the
> > implementation phase there are some obstacles that are not easy to
> > overcome.
>
> Yeah, I agree that this should be recommended rather than enforced via
> some checkstyle rule. However, reviewers should be aware of this
> recommendation in the code-style page.
>
> >
> > So, adding the MissingDeprecated will emphasize to a developer the
> > need to describe the deprecation reasons in comments, but
> > unfortunately, there is no general pattern that we can enforce for
> > every such description message and/or automatically validate its
> > meaningfulness. There may be no alternative for a deprecated field, or
> > it may simply be marked for deletion, so the pattern is slightly
> > different in this case.
>
>
> +1 for adding the MissingDeprecated rule
>
> > Another problem is how to add meaningful comments to the deprecated
> > annotations that we already have in the code, since we can't enforce
> > checkstyle rules only on newly added code. This is a very exhausting
> > process with no 100% guarantee of accuracy - some of the commits don't
> > have a good commit message and require a deep archaeology.
>
> Not aiming for 100% accuracy, but more on code style agreement.
>
> > All of the above led me to the following which is pretty easy to
> > achieve and improves the code quality:
> >
> > /** @deprecated See CASSANDRA-6504 */
> > @Deprecated(since = "2.1")
> > public Integer concurrent_replicates = null;
> >
> > On Wed, 11 Oct 2023 at 09:51, Miklosovic, Stefan
> >  wrote:
> > >
> > > Here (1) it supports check of both Javadoc and annotation at the same
> time so what you want is possible. What is not possible is to checkstyle
> the _content_ of deprecated Javadoc nor any format of it. I think that
> ensuring the presence of both annotation and Javadoc comment is just enough.
> > >
> > > (1)
> https://checkstyle.sourceforge.io/apidocs/com/puppycrawl/tools/checkstyle/checks/annotation/MissingDeprecatedCheck.html
> > >
> > > 
> > > From: Francisco Guerrero 
> > > Sent: Tuesday, October 10, 2023 23:34
> > > To: dev@cassandra.apache.org
> > > Subject: Re: [DISCUSS] putting versions into Deprecated annotations
> > >
> > > NetApp Security WARNING: This is an external email. Do not click links
> or open attachments unless you recognize the sender and know the content is
> safe.
> > >
> > >
> > >
> > >
> > > To me this seems insufficient. As a developer, I'd like to see what
> the alternative is when reading the javadoc without having to go to Jira.
> > >
> > > What I would prefer is to know what the alternative is and how to use
> it. For example:
> > >
> > > /** @deprecated Use {@link #alternative} instead. See CASSANDRA-6504 */
> > > @Deprecated(since = "2.1")
> > > public Integer concurrent_replicates = null;
> > >
> > > I am not sure if checkstyle can enforce the above, so the mechanisms
> to enforce it would still need to be laid out, unless we can easily support
> something like the above with checkstyle rules.
> > >
> > > On 2023/10/10 20:34:27 Maxim Muzafarov wrote:
> > > > Hello everyone,
> > > >
> > > >
> > > > I've discussed with Stefan some steps we can take to improve the
> final
> > > > solution, so the final version might look like this:
> > > >
> > > > /** @deprecated See CASSANDRA-6504 */
> > > > @Deprecated(since = "2.1")
> > > > public Integer concurrent_replicates = null;
> > > >
> > > > The issue number will be taken from the git blame comment. I doubt I
> > > > can generate and/or create a meaningful comment for every deprecation
> > > > annotation, but providing a link to the issue that was retrieved from
> > > > the git blame is not too big a problem. This also improves the
> > > > visibility.
> > > >
> > > > In addition, we can add two checkstyle rules [1] [2] to ensure that
> > > > any future deprecations will have a "since" element and a JavaDoc
> > > > comment.
> > > > WDYT?
> > > >
> > > > [1]
> https://checkstyle.sourceforge.io/apidocs/com/puppycrawl/tools/checkstyle/checks/annotation/MissingDeprecatedCheck.html
> > > > [2]
> https://checkstyle.org/apidocs/com/puppycrawl/tools/checkstyle/checks/coding/MatchXpathCheck.html
> > > >
> > > > On Tue, 10 Oct 2023 at 14:50, Josh McKenzie 
> wrote:
> > > > >
> > > > > Sounds like we're relitigating the basics of how @Deprecated,
> forRemoval, since, and javadoc @link all intersect to make deprecation less
> painful ;)
> > > > >
> > > > > So:
> > > > >
> > > > > Buil

Re: [DISCUSS] putting versions into Deprecated annotations

[Benjamin Lerer]

I was asking because outside of configuration parameters and JMX calls, the
approach as far as I remember was to just change things without using an
annotation.

Le ven. 13 oct. 2023 à 12:45, Miklosovic, Stefan via dev <
dev@cassandra.apache.org> a écrit :

> Hi Benjamin,
>
> in other words, anything we have @Deprecated annotation on top of (or
> anything you want to annotate with it). Does it help with the explanation?
>
> For the initial phase, I plan to just put "since" everywhere (into every
> already existing @Deprecated annotation) and we leave out "forRemoval" in
> Deprecated annotation for now as that is quite tricky to get right.
>
> I am confused what is considered to be removed and what we keep there for
> ever even it is deprecated (referring to what Mick said in this thread that
> forRemoval can not be by default true). After we map what technical debt we
> have, we can summarize this and I bring it to the ML again for further
> discussion what to actually remove and when.
>
> Regards
>
> 
> From: Benjamin Lerer 
> Sent: Friday, October 13, 2023 12:19
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] putting versions into Deprecated annotations
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I am a bit confused by the starting point of this discussion: "When we
> deprecate APIs / methods"
> What are we exactly calling APIs/methods? It is really unclear to me what
> we are talking about here.
>
> Le jeu. 12 oct. 2023 à 02:38, Francisco Guerrero  <mailto:fran...@apache.org>> a écrit :
>
>
> On 2023/10/11 16:59:35 Maxim Muzafarov wrote:
> > Francisco,
> >
> > I agree with your vision of the deprecation comments and actually, I
> > think we should recommend doing it that way for the cases where it is
> > applicable on our code-style page, but when things get to the
> > implementation phase there are some obstacles that are not easy to
> > overcome.
>
> Yeah, I agree that this should be recommended rather than enforced via
> some checkstyle rule. However, reviewers should be aware of this
> recommendation in the code-style page.
>
> >
> > So, adding the MissingDeprecated will emphasize to a developer the
> > need to describe the deprecation reasons in comments, but
> > unfortunately, there is no general pattern that we can enforce for
> > every such description message and/or automatically validate its
> > meaningfulness. There may be no alternative for a deprecated field, or
> > it may simply be marked for deletion, so the pattern is slightly
> > different in this case.
>
>
> +1 for adding the MissingDeprecated rule
>
> > Another problem is how to add meaningful comments to the deprecated
> > annotations that we already have in the code, since we can't enforce
> > checkstyle rules only on newly added code. This is a very exhausting
> > process with no 100% guarantee of accuracy - some of the commits don't
> > have a good commit message and require a deep archaeology.
>
> Not aiming for 100% accuracy, but more on code style agreement.
>
> > All of the above led me to the following which is pretty easy to
> > achieve and improves the code quality:
> >
> > /** @deprecated See CASSANDRA-6504 */
> > @Deprecated(since = "2.1")
> > public Integer concurrent_replicates = null;
> >
> > On Wed, 11 Oct 2023 at 09:51, Miklosovic, Stefan
> > mailto:stefan.mikloso...@netapp.com>>
> wrote:
> > >
> > > Here (1) it supports check of both Javadoc and annotation at the same
> time so what you want is possible. What is not possible is to checkstyle
> the _content_ of deprecated Javadoc nor any format of it. I think that
> ensuring the presence of both annotation and Javadoc comment is just enough.
> > >
> > > (1)
> https://checkstyle.sourceforge.io/apidocs/com/puppycrawl/tools/checkstyle/checks/annotation/MissingDeprecatedCheck.html
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcheckstyle.sourceforge.io%2Fapidocs%2Fcom%2Fpuppycrawl%2Ftools%2Fcheckstyle%2Fchecks%2Fannotation%2FMissingDeprecatedCheck.html&data=05%7C01%7CStefan.Miklosovic%40netapp.com%7C59fa2b3786ff436c83ba08dbcbd5ece7%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638327891917050879%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=8qKu8ob%2BvPdHfUQdkxr5C%2BgkR5iMcUaEqw9a%2FNN276k%3D&reserved=0
> >
> > >
> > > ___

Re: [DISCUSS] putting versions into Deprecated annotations

[Benjamin Lerer]

>
> Could you point me some document / ML thread this was explicitly decided
> in if you know of anything like that? It would be great if there was some
> solid guidance on this.


I am seeing it the other way around. Using Deprecated annotations make
sense only if something is part of a public interface/API. Maintaining a
public API represent a significant work and put some constraints on further
evolution.
By default most of the code of C* should be considered as internal and we
should be able to modify it without going through a deprecation phase.
One problem that we have is that we have never been clear, outside of some
obvious stuff, about what code should be consider as APIs that need to go
through a deprecation phase.


Le ven. 13 oct. 2023 à 13:13, Miklosovic, Stefan via dev <
dev@cassandra.apache.org> a écrit :

> OK. That is definitely something to mention when we will approach the
> second phase where  we decide what do with it but I humbly think we are not
> there yet.
>
> Could you point me some document / ML thread this was explicitly decided
> in if you know of anything like that? It would be great if there was some
> solid guidance on this.
>
> ____
> From: Benjamin Lerer 
> Sent: Friday, October 13, 2023 13:07
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] putting versions into Deprecated annotations
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I was asking because outside of configuration parameters and JMX calls,
> the approach as far as I remember was to just change things without using
> an annotation.
>
> Le ven. 13 oct. 2023 à 12:45, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>> a écrit :
> Hi Benjamin,
>
> in other words, anything we have @Deprecated annotation on top of (or
> anything you want to annotate with it). Does it help with the explanation?
>
> For the initial phase, I plan to just put "since" everywhere (into every
> already existing @Deprecated annotation) and we leave out "forRemoval" in
> Deprecated annotation for now as that is quite tricky to get right.
>
> I am confused what is considered to be removed and what we keep there for
> ever even it is deprecated (referring to what Mick said in this thread that
> forRemoval can not be by default true). After we map what technical debt we
> have, we can summarize this and I bring it to the ML again for further
> discussion what to actually remove and when.
>
> Regards
>
> 
> From: Benjamin Lerer mailto:b.le...@gmail.com>>
> Sent: Friday, October 13, 2023 12:19
> To: dev@cassandra.apache.org<mailto:dev@cassandra.apache.org>
> Subject: Re: [DISCUSS] putting versions into Deprecated annotations
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> I am a bit confused by the starting point of this discussion: "When we
> deprecate APIs / methods"
> What are we exactly calling APIs/methods? It is really unclear to me what
> we are talking about here.
>
> Le jeu. 12 oct. 2023 à 02:38, Francisco Guerrero  <mailto:fran...@apache.org><mailto:fran...@apache.org fran...@apache.org>>> a écrit :
>
>
> On 2023/10/11 16:59:35 Maxim Muzafarov wrote:
> > Francisco,
> >
> > I agree with your vision of the deprecation comments and actually, I
> > think we should recommend doing it that way for the cases where it is
> > applicable on our code-style page, but when things get to the
> > implementation phase there are some obstacles that are not easy to
> > overcome.
>
> Yeah, I agree that this should be recommended rather than enforced via
> some checkstyle rule. However, reviewers should be aware of this
> recommendation in the code-style page.
>
> >
> > So, adding the MissingDeprecated will emphasize to a developer the
> > need to describe the deprecation reasons in comments, but
> > unfortunately, there is no general pattern that we can enforce for
> > every such description message and/or automatically validate its
> > meaningfulness. There may be no alternative for a deprecated field, or
> > it may simply be marked for deletion, so the pattern is slightly
> > different in this case.
>
>
> +1 for adding the MissingDeprecated rule
>
> > Another problem is how to add meaningful comments to the deprecated
> > annotations that we already have in the code, sin

Re: [DISCUSS] putting versions into Deprecated annotations

[Benjamin Lerer]

>
> I’ve been told in the past not to remove public methods in a patch release
> though.
>

Then I am curious to get the rationale behind that. If some piece of code
is not used anymore then simplifying the code is the best thing to do. It
makes maintenance easier and avoids mistakes.
Le ven. 13 oct. 2023 à 14:11, Miklosovic, Stefan via dev <
dev@cassandra.apache.org> a écrit :

> Maybe for better understanding what we talk about, there is the PR which
> implements the changes suggested here (1)
>
> It is clear that @Deprecated is not used exclusively on JMX /
> Configuration but we use it internally as well. This is a very delicate
> topic and we need to go, basically, one by one.
>
> I get that there might be some kind of a "nervousness" around this as we
> strive for not breaking it unnecessarily so there might be a lot of
> exceptions etc and I completely understand that but what I lack is clear
> visibility into what we plan to do with it (if anything).
>
> There is deprecated stuff as old as Cassandra 1.2 / 2.0 (!!!) and it is
> really questionable if we should not just get rid of that once for all. I
> am OK with keeping it there if we decide that, but we should provide some
> additional information like when it was deprecated and why it is necessary
> to keep it around otherwise the code-base will bloat and bloat ...
>
> (1) https://github.com/apache/cassandra/pull/2801/files
>
> 
> From: Mick Semb Wever 
> Sent: Friday, October 13, 2023 13:51
> To: dev@cassandra.apache.org
> Subject: Re: [DISCUSS] putting versions into Deprecated annotations
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
>
> On Fri, 13 Oct 2023 at 13:07, Benjamin Lerer  ble...@apache.org>> wrote:
> I was asking because outside of configuration parameters and JMX calls,
> the approach as far as I remember was to just change things without using
> an annotation.
>
>
> Yes, it is my understanding that such deprecation is only needed on
> methods/objects that belong to some API/SPI component of ours that requires
> compatibility.  This is much more than configuration and JMX, and can be
> quite subtle in areas.   A failed attempt I started at this is here:
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28wip%29+Compatibility+Planning
> <
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCASSANDRA%2F%2528wip%2529%2BCompatibility%2BPlanning&data=05%7C01%7CStefan.Miklosovic%40netapp.com%7Cf0af5e7db5e9468faa8c08dbcbe2e9f3%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638327947670748135%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3363U2ZlI%2FasNF0NTMrdZ%2BogB%2FjigmCGt3zqRrIs99Q%3D&reserved=0
> >
>
> But there will also be internal methods/objects marked as deprecated that
> relate back to these compatibility concerns, annotated because their
> connection and removal might not be so obvious when the time comes.
>

Re: [DISCUSS] putting versions into Deprecated annotations

[Benjamin Lerer]

Ok, thanks Stefan I understand the context better now. Looking at the PR.
Some make sense also for serialization reasons but some make no sense to me.


Le ven. 13 oct. 2023 à 14:26, Benjamin Lerer  a écrit :

> I’ve been told in the past not to remove public methods in a patch release
>> though.
>>
>
> Then I am curious to get the rationale behind that. If some piece of code
> is not used anymore then simplifying the code is the best thing to do. It
> makes maintenance easier and avoids mistakes.
> Le ven. 13 oct. 2023 à 14:11, Miklosovic, Stefan via dev <
> dev@cassandra.apache.org> a écrit :
>
>> Maybe for better understanding what we talk about, there is the PR which
>> implements the changes suggested here (1)
>>
>> It is clear that @Deprecated is not used exclusively on JMX /
>> Configuration but we use it internally as well. This is a very delicate
>> topic and we need to go, basically, one by one.
>>
>> I get that there might be some kind of a "nervousness" around this as we
>> strive for not breaking it unnecessarily so there might be a lot of
>> exceptions etc and I completely understand that but what I lack is clear
>> visibility into what we plan to do with it (if anything).
>>
>> There is deprecated stuff as old as Cassandra 1.2 / 2.0 (!!!) and it is
>> really questionable if we should not just get rid of that once for all. I
>> am OK with keeping it there if we decide that, but we should provide some
>> additional information like when it was deprecated and why it is necessary
>> to keep it around otherwise the code-base will bloat and bloat ...
>>
>> (1) https://github.com/apache/cassandra/pull/2801/files
>>
>> 
>> From: Mick Semb Wever 
>> Sent: Friday, October 13, 2023 13:51
>> To: dev@cassandra.apache.org
>> Subject: Re: [DISCUSS] putting versions into Deprecated annotations
>>
>> NetApp Security WARNING: This is an external email. Do not click links or
>> open attachments unless you recognize the sender and know the content is
>> safe.
>>
>>
>>
>>
>>
>> On Fri, 13 Oct 2023 at 13:07, Benjamin Lerer > ble...@apache.org>> wrote:
>> I was asking because outside of configuration parameters and JMX calls,
>> the approach as far as I remember was to just change things without using
>> an annotation.
>>
>>
>> Yes, it is my understanding that such deprecation is only needed on
>> methods/objects that belong to some API/SPI component of ours that requires
>> compatibility.  This is much more than configuration and JMX, and can be
>> quite subtle in areas.   A failed attempt I started at this is here:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/%28wip%29+Compatibility+Planning
>> <
>> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCASSANDRA%2F%2528wip%2529%2BCompatibility%2BPlanning&data=05%7C01%7CStefan.Miklosovic%40netapp.com%7Cf0af5e7db5e9468faa8c08dbcbe2e9f3%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638327947670748135%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=3363U2ZlI%2FasNF0NTMrdZ%2BogB%2FjigmCGt3zqRrIs99Q%3D&reserved=0
>> >
>>
>> But there will also be internal methods/objects marked as deprecated that
>> relate back to these compatibility concerns, annotated because their
>> connection and removal might not be so obvious when the time comes.
>>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

My biggest concern with the 5.1 suggestion is that it makes the review of
TCM far more complicated than it should be. Even if all TCM patches were
fully reviewed by committers that I fully trust, due to the patch size and
the impact of the changes it feels safer to me to have a second round of
review now than the dust is settling. Merging TCM and Accord in a 5.1
branch and doing the review there will make things harder than it should.

Le lun. 23 oct. 2023 à 23:02, Dinesh Joshi  a écrit :

> I have a strong preference to move out the 5.0 date to have accord and
> TCM. I don’t see the point in shipping 5.0 without these features
> especially if 5.1 is going to follow close behind it.
>
> Dinesh
>
> On Oct 23, 2023, at 4:52 AM, Mick Semb Wever  wrote:
>
> 
>
> The TCM work (CEP-21) is in its review stage but being well past our
> cut-off date¹ for merging, and now jeopardising 5.0 GA efforts, I would
> like to propose the following.
>
> We merge TCM and Accord only to trunk.  Then branch cassandra-5.1 and cut
> an immediate 5.1-alpha1 release.
>
> I see this as a win-win scenario for us, considering our current
> situation.  (Though it is unfortunate that Accord is included in this
> scenario because we agreed it to be based upon TCM.)
>
> This will mean…
>  - We get to focus on getting 5.0 to beta and GA, which already has a ton
> of features users want.
>  - We get an alpha release with TCM and Accord into users hands quickly
> for broader testing and feedback.
>  - We isolate GA efforts on TCM and Accord – giving oss and downstream
> engineers time and patience reviewing and testing.  TCM will be the biggest
> patch ever to land in C*.
>  - Give users a choice for a more incremental upgrade approach, given just
> how many new features we're putting on them in one year.
>  - 5.1 w/ TCM and Accord will maintain its upgrade compatibility with all
> 4.x versions, just as if it had landed in 5.0.
>
>
> The risks/costs this introduces are
>  - If we cannot stabilise TCM and/or Accord on the cassandra-5.1 branch,
> and at some point decide to undo this work, while we can throw away the
> cassandra-5.1 branch we would need to do a bit of work reverting the
> changes in trunk.  This is a _very_ edge case, as confidence levels on the
> design and implementation of both are already tested and high.
>  - We will have to maintain an additional branch.  I propose that we treat
> the 5.1 branch in the same maintenance window as 5.0 (like we have with 3.0
> and 3.11).  This also adds the merge path overhead.
>  - Reviewing of TCM and Accord will continue to happen post-merge.  This
> is not our normal practice, but this work will have already received its
> two +1s from committers, and such ongoing review effort is akin to GA
> stabilisation work on release branches.
>
>
> I see no other ok solution in front of us that gets us at least both the
> 5.0 beta and TCM+Accord alpha releases this year.  Keeping in mind users
> demand to start experimenting with these features, and our Cassandra Summit
> in December.
>
>
> 1) https://lists.apache.org/thread/9c5cnn57c7oqw8wzo3zs0dkrm4f17lm3
>
>
>

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

The proposal includes 3 things:
1. Do not include TCM and Accord in 5.0 to avoid delaying 5.0
2. The next release will be 5.1 and will include only Accord and TCM
3. Merge TCM and Accord right now in 5.1 (making an initial release)

I am fine with question 1 and do not have a strong opinion on which way to
go.
2. Means that every new feature will have to wait for post 5.1 even if it
is ready before 5.1 is stabilized and shipped. If we do a 5.1 release why
not take it as an opportunity to release more things. I am not saying that
we will. Just that we should let that door open.
3. There is a need to merge TCM and Accord as maintaining those separate
branches is costly in terms of time and energy. I fully understand that. On
the other hand merging TCM and Accord will make the TCM review harder and I
do believe that this second round of review is valuable as it already
uncovered a valid issue. Nevertheless, I am fine with merging TCM as soon
as it passes CI and continuing the review after the merge. If we cannot
meet at least that quality level (Green CI) we should not merge just for
creating an 5.1.alpha release for the summit.

Now, I am totally fine with a preview release without numbering and with
big warnings that will only serve as a preview for the summit.

Le mer. 25 oct. 2023 à 06:33, Berenguer Blasi  a
écrit :

> I also think there's many good new features in 5.0 already they'd make a
> good release even on their own. My 2 cts.
>
> On 24/10/23 23:20, Brandon Williams wrote:
> > The catch here is that we don't publish docker images currently.  The
> > C* docker images available are not made by us.
> >
> > Kind Regards,
> > Brandon
> >
> > On Tue, Oct 24, 2023 at 3:31 PM Patrick McFadin 
> wrote:
> >> Let me make that really easy. Hell yes
> >>
> >> Not everybody runs CCM, I've tried but I've met resistance.
> >>
> >> Compiling your own version usually involves me saying the words "Yes,
> ant realclean exists. I'm not trolling you"
> >>
> >> docker pull  works on every OS and curates a single node
> experience.
> >>
> >>
> >>
> >> On Tue, Oct 24, 2023 at 12:37 PM Josh McKenzie 
> wrote:
> >>> In order for the project to advertise the release outside the dev@
> list it needs to be a formal release.
> >>>
> >>> That's my reading as well:
> >>> https://www.apache.org/legal/release-policy.html#release-definition
> >>>
> >>> I wonder if there'd be value in us having a cronned job that'd do
> nightly docker container builds on trunk + feature branches, archived for N
> days, and we make that generally known to the dev@ list here so folks
> that want to poke at the current state of trunk or other branches could do
> so with very low friction. We'd probably see more engagement on feature
> branches if it was turn-key easy for other C* devs to spin the up and check
> them out.
> >>>
> >>> For what you're talking about here Patrick (a docker image for folks
> outside the dev@ audience and more user-facing), we'd want to vote on it
> and go through the formal process.
> >>>
> >>> On Tue, Oct 24, 2023, at 3:10 PM, Jeremiah Jordan wrote:
> >>>
> >>> In order for the project to advertise the release outside the dev@
> list it needs to be a formal release.  That just means that there was a
> release vote and at least 3 PMC members +1’ed it, and there are more +1
> than there are -1, and we follow all the normal release rules.  The ASF
> release process doesn’t care what branch you cut the artifacts from or what
> version you call it.
> >>>
> >>> So the project can cut artifacts for and release a 5.1-alpha1,
> 5.1-dev-preview1, what ever we want to version this thing, from trunk or
> any other branch name we want.
> >>>
> >>> -Jeremiah
> >>>
> >>> On Oct 24, 2023 at 2:03:41 PM, Patrick McFadin 
> wrote:
> >>>
> >>> I would like to have something for developers to use ASAP to try the
> Accord syntax. Very few people have seen it, and I think there's a learning
> curve we can start earlier.
> >>>
> >>> It's my understanding that ASF policy is that it needs to be a project
> release to create a docker image.
> >>>
> >>> On Tue, Oct 24, 2023 at 11:54 AM Jeremiah Jordan <
> jeremiah.jor...@gmail.com> wrote:
> >>>
> >>> If we decide to go the route of not merging TCM to the 5.0 branch.  Do
> we actually need to immediately cut a 5.1 branch?  Can we work on
> stabilizing things while it is in trunk and cut the 5.1 branch when we
> actually think we are near releasing?  I don’t see any reason we can not
> cut “preview” artifacts from trunk?
> >>>
> >>> -Jeremiah
> >>>
> >>> On Oct 24, 2023 at 11:54:25 AM, Jon Haddad 
> wrote:
> >>>
> >>> I guess at the end of the day, shipping a release with a bunch of
> awesome features is better than holding it back.  If there's 2 big releases
> in 6 months the community isn't any worse off.
> >>>
> >>> We either ship something, or nothing, and something is probably better.
> >>>
> >>> Jon
> >>>
> >>>
> >>> On 2023/10/24 16:27:04 Patrick McFadin wrote:
> >>>
> >>> +1 to what you are saying

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

>
> I am surprised this needs to be said, but - especially for long-running
> CEPs - you must involve yourself early, and certainly within some
> reasonable time of being notified the work is ready for broader input and
> review. In this case, more than six months ago.


It is unfortunately more complicated than that because six month ago
Ekaterina and I were working on supporting Java 17 and dropping Java 8
which was needed by different ongoing works. We both missed the
announcement that TCM was ready for review and anyway would not have been
available at that time. Maxim has asked me ages ago for a review of
CASSANDRA-15254 <https://issues.apache.org/jira/browse/CASSANDRA-15254>
more than 6 months ago and I have not been able to help him so far. We all
have a limited bandwidth and can miss some announcements.

The project has grown and a lot of things are going on in parallel. There
are also more interdependencies between the different projects. In my
opinion what we are lacking is a global overview of the different things
going on in the project and some rough ideas of the status of the different
significant pieces. It would allow us to better organize ourselves.

Le jeu. 26 oct. 2023 à 00:26, Benedict  a écrit :

> I have spoken privately with Ekaterina, and to clear up some possible
> ambiguity: I realise nobody has demanded a delay to this work to conduct
> additional reviews; a couple of folk have however said they would prefer
> one.
>
>
> My point is that, as a community, we need to work on ensuring folk that
> care about a CEP participate at an appropriate time. If they aren’t able
> to, the consequences of that are for them to bear.
>
>
> We should be working to avoid surprises as CEP start to land. To this end,
> I think we should work on some additional paragraphs for the governance doc
> covering expectations around the landing of CEPs.
>
> On 25 Oct 2023, at 21:55, Benedict  wrote:
>
> 
>
> I am surprised this needs to be said, but - especially for long-running
> CEPs - you must involve yourself early, and certainly within some
> reasonable time of being notified the work is ready for broader input and
> review. In this case, more than six months ago.
>
>
> This isn’t the first time this has happened, and it is disappointing to
> see it again. Clearly we need to make this explicit in the guidance docs.
>
>
> Regarding the release of 5.1, I understood the proposal to be that we cut
> an actual alpha, thereby sealing the 5.1 release from new features. Only
> features merged before we cut the alpha would be permitted, and the alpha
> should be cut as soon as practicable. What exactly would we be waiting for?
>
>
> If we don’t have a clear and near-term trigger for branching 5.1 for its
> own release, shortly after Accord and TCM merge, then I am in favour of
> instead delaying 5.0.
>
> On 25 Oct 2023, at 19:40, Mick Semb Wever  wrote:
>
> 
> I'm open to the suggestions of not branching cassandra-5.1 and/or naming a
> preview release something other than 5.1-alpha1.
>
> But… the codebases and release process (and upgrade tests) do not
> currently support releases with qualifiers that are not alpha, beta, or
> rc.  We can add a preview qualifier to this list, but I would not want to
> block getting a preview release out only because this wasn't yet in place.
>
> Hence the proposal used 5.1-alpha1 simply because that's what we know we
> can do today.  An alpha release also means (typically) the branch.
>
> Is anyone up for looking into adding a "preview" qualifier to our release
> process?
> This may also solve our previous discussions and desire to have quarterly
> releases that folk can use through the trunk dev cycle.
>
> Personally, with my understanding of timelines in front of us to fully
> review and stabilise tcm, it makes sense to branch it as soon as it's
> merged.  It's easiest to stabilise it on a branch, and there's definitely
> the desire and demand to do so, so it won't be getting forgotten or
> down-prioritised.
>
>
>
> On Wed, 25 Oct 2023 at 18:07, Jeremiah Jordan 
> wrote:
>
>> If we do a 5.1 release why not take it as an opportunity to release more
>>> things. I am not saying that we will. Just that we should let that door
>>> open.
>>>
>>
>> Agreed.  This is the reason I brought up the possibility of not branching
>> off 5.1 immediately.
>>
>>
>> On Oct 25, 2023 at 3:17:13 AM, Benjamin Lerer  wrote:
>>
>>> The proposal includes 3 things:
>>> 1. Do not include TCM and Accord in 5.0 to avoid delaying 5.0
>>> 2. The next release will be 5.1 and will include only Accord and TCM
>>> 3. Merge TCM and 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

>
> Regarding the release of 5.1, I understood the proposal to be that we cut
> an actual alpha, thereby sealing the 5.1 release from new features. Only
> features merged before we cut the alpha would be permitted, and the alpha
> should be cut as soon as practicable. What exactly would we be waiting for?


The problem I believe is about expectations. It seems that your expectation
is that a release with only TCM and Accord will reach GA quickly. Based on
the time it took us to release 4.1, I am simply expecting more delays (a GA
around end of May, June). In which case it seems to me that we could be
interested in shipping more stuff in the meantime (thinking of
CASSANDRA-15254 or CEP-29 for example).
I do not have a strong opinion, I just want to make sure that we all share
the same understanding and fully understand what we agree upon.

Le jeu. 26 oct. 2023 à 10:59, Benjamin Lerer  a écrit :

> I am surprised this needs to be said, but - especially for long-running
>> CEPs - you must involve yourself early, and certainly within some
>> reasonable time of being notified the work is ready for broader input and
>> review. In this case, more than six months ago.
>
>
> It is unfortunately more complicated than that because six month ago
> Ekaterina and I were working on supporting Java 17 and dropping Java 8
> which was needed by different ongoing works. We both missed the
> announcement that TCM was ready for review and anyway would not have been
> available at that time. Maxim has asked me ages ago for a review of
> CASSANDRA-15254 <https://issues.apache.org/jira/browse/CASSANDRA-15254>
> more than 6 months ago and I have not been able to help him so far. We all
> have a limited bandwidth and can miss some announcements.
>
> The project has grown and a lot of things are going on in parallel. There
> are also more interdependencies between the different projects. In my
> opinion what we are lacking is a global overview of the different things
> going on in the project and some rough ideas of the status of the different
> significant pieces. It would allow us to better organize ourselves.
>
> Le jeu. 26 oct. 2023 à 00:26, Benedict  a écrit :
>
>> I have spoken privately with Ekaterina, and to clear up some possible
>> ambiguity: I realise nobody has demanded a delay to this work to conduct
>> additional reviews; a couple of folk have however said they would prefer
>> one.
>>
>>
>> My point is that, as a community, we need to work on ensuring folk that
>> care about a CEP participate at an appropriate time. If they aren’t able
>> to, the consequences of that are for them to bear.
>>
>>
>> We should be working to avoid surprises as CEP start to land. To this
>> end, I think we should work on some additional paragraphs for the
>> governance doc covering expectations around the landing of CEPs.
>>
>> On 25 Oct 2023, at 21:55, Benedict  wrote:
>>
>> 
>>
>> I am surprised this needs to be said, but - especially for long-running
>> CEPs - you must involve yourself early, and certainly within some
>> reasonable time of being notified the work is ready for broader input and
>> review. In this case, more than six months ago.
>>
>>
>> This isn’t the first time this has happened, and it is disappointing to
>> see it again. Clearly we need to make this explicit in the guidance docs.
>>
>>
>> Regarding the release of 5.1, I understood the proposal to be that we cut
>> an actual alpha, thereby sealing the 5.1 release from new features. Only
>> features merged before we cut the alpha would be permitted, and the alpha
>> should be cut as soon as practicable. What exactly would we be waiting for?
>>
>>
>> If we don’t have a clear and near-term trigger for branching 5.1 for its
>> own release, shortly after Accord and TCM merge, then I am in favour of
>> instead delaying 5.0.
>>
>> On 25 Oct 2023, at 19:40, Mick Semb Wever  wrote:
>>
>> 
>> I'm open to the suggestions of not branching cassandra-5.1 and/or naming
>> a preview release something other than 5.1-alpha1.
>>
>> But… the codebases and release process (and upgrade tests) do not
>> currently support releases with qualifiers that are not alpha, beta, or
>> rc.  We can add a preview qualifier to this list, but I would not want to
>> block getting a preview release out only because this wasn't yet in place.
>>
>> Hence the proposal used 5.1-alpha1 simply because that's what we know we
>> can do today.  An alpha release also means (typically) the branch.
>>
>> Is anyone up for looking into adding a "preview" qualifier to our relea

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

with regular updates, could help
> create a real sense of the remaining work in progress. These are my
> personal feelings, and definitely not actions to be taken. The example
> is here: [2].
>
> [2]
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Flists.apache.org%2Fthread%2Fm11m0nxq701f2cj8xxdcsc4nnn2sm8ql&data=05%7C01%7CGerman.Eichberger%40microsoft.com%7Cc811f6a430d1466acc3f08dbd61639c2%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638339163187360611%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=%2BG0wgMItsMv83XDLzRgbfJoi%2FiwSywWU0qAzN%2BmMBZU%3D&reserved=0
> <https://lists.apache.org/thread/m11m0nxq701f2cj8xxdcsc4nnn2sm8ql>
>
> On Thu, 26 Oct 2023 at 11:15, Benjamin Lerer  wrote:
> >>
> >> Regarding the release of 5.1, I understood the proposal to be that we
> cut an actual alpha, thereby sealing the 5.1 release from new features.
> Only features merged before we cut the alpha would be permitted, and the
> alpha should be cut as soon as practicable. What exactly would we be
> waiting for?
> >
> >
> > The problem I believe is about expectations. It seems that your
> expectation is that a release with only TCM and Accord will reach GA
> quickly. Based on the time it took us to release 4.1, I am simply expecting
> more delays (a GA around end of May, June). In which case it seems to me
> that we could be interested in shipping more stuff in the meantime
> (thinking of CASSANDRA-15254 or CEP-29 for example).
> > I do not have a strong opinion, I just want to make sure that we all
> share the same understanding and fully understand what we agree upon.
> >
> > Le jeu. 26 oct. 2023 à 10:59, Benjamin Lerer  a
> écrit :
> >>>
> >>> I am surprised this needs to be said, but - especially for
> long-running CEPs - you must involve yourself early, and certainly within
> some reasonable time of being notified the work is ready for broader input
> and review. In this case, more than six months ago.
> >>
> >>
> >> It is unfortunately more complicated than that because six month ago
> Ekaterina and I were working on supporting Java 17 and dropping Java 8
> which was needed by different ongoing works. We both missed the
> announcement that TCM was ready for review and anyway would not have been
> available at that time. Maxim has asked me ages ago for a review of
> CASSANDRA-15254  more than 6 months ago and I have not been able to help
> him so far. We all have a limited bandwidth and can miss some announcements.
> >>
> >> The project has grown and a lot of things are going on in parallel.
> There are also more interdependencies between the different projects. In my
> opinion what we are lacking is a global overview of the different things
> going on in the project and some rough ideas of the status of the different
> significant pieces. It would allow us to better organize ourselves.
> >>
> >> Le jeu. 26 oct. 2023 à 00:26, Benedict  a écrit :
> >>>
> >>> I have spoken privately with Ekaterina, and to clear up some possible
> ambiguity: I realise nobody has demanded a delay to this work to conduct
> additional reviews; a couple of folk have however said they would prefer
> one.
> >>>
> >>>
> >>> My point is that, as a community, we need to work on ensuring folk
> that care about a CEP participate at an appropriate time. If they aren’t
> able to, the consequences of that are for them to bear.
> >>>
> >>>
> >>> We should be working to avoid surprises as CEP start to land. To this
> end, I think we should work on some additional paragraphs for the
> governance doc covering expectations around the landing of CEPs.
> >>>
> >>>
> >>> On 25 Oct 2023, at 21:55, Benedict  wrote:
> >>>
> >>> 
> >>>
> >>> I am surprised this needs to be said, but - especially for
> long-running CEPs - you must involve yourself early, and certainly within
> some reasonable time of being notified the work is ready for broader input
> and review. In this case, more than six months ago.
> >>>
> >>>
> >>> This isn’t the first time this has happened, and it is disappointing
> to see it again. Clearly we need to make this explicit in the guidance docs.
> >>>
> >>>
> >>> Regarding the release of 5.1, I understood the proposal to be that we
> cut an actual alpha, thereby sealing the 5.1 release from new features.
> Only features merged before we cut the alpha would be permitted, and the
> alpha should be cut as soon as practicable. Wha

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

r authors, and writing Harry
> tests. I would not, however, ask someone to postpone a feature based on my
> past or future availability.
>
> On Fri, Oct 27, 2023, at 10:14 AM, Jacek Lewandowski wrote:
>
> I've been thinking about this and I believe that if we ever decide to
> delay a release to include some CEPs, we should make the plan and status of
> those CEPs public. This should include publishing a branch, creating
> tickets for the remaining work required for feature completion in Jira, and
> notifying the mailing list.
>
> By doing this, we can make an informed decision about whether delivering a
> CEP in a release x.y planned for some time z is feasible. This approach
> would also be beneficial for improving collaboration, as we will all be
> aware of what is left to be done and can adjust our focus accordingly to
> participate in the remaining work.
>
> Thanks,
> - - -- --- -  -
> Jacek Lewandowski
>
>
> pt., 27 paź 2023 o 10:26 Benjamin Lerer  napisał(a):
>
> I would be interested in testing Maxim's approach. We need more visibility
> on big features and their progress to improve our coordination. Hopefully
> it will also open the door to more collaboration on those big projects.
>
> Le jeu. 26 oct. 2023 à 21:35, German Eichberger via dev <
> dev@cassandra.apache.org> a écrit :
>
> +1 to Maxim's idea
>
> Like Stefan my assumption was that we would get some version of TCM +
> ACCORD in 5.0 but it wouldn't be ready for production use. My own testing
> and conversations at Community over Code in Halifax confirmed this.
>
> From this perspective as disappointing as TCM+ACCORD slipping is moving it
> to 5.1 makes sense and I am supporting of this - but I am worried if 5.1 is
> basically 5.0 + TCM/ACCORD and this slips again we draw ourselves into a
> corner where we can't release 5.2 before 5.1 or something. I would like
> some more elaboration on that.
>
> I am also very worried about ANN vector search being in jeopardy for 5.0
> which is an important feature for me to win some internal company bet 🙂
>
> My 2 cents,
> German
>
> --
>
>
> *From:* Miklosovic, Stefan via dev 
> *Sent:* Thursday, October 26, 2023 4:23 AM
> *To:* dev@cassandra.apache.org 
> *Cc:* Miklosovic, Stefan 
> *Subject:* [EXTERNAL] Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1
> (and cut an immediate 5.1-alpha1)
>
> What Maxim proposes in the last paragraph would be definitely helpful. Not
> for the project only but for a broader audience, companies etc., too.
>
> Until this thread was started, my assumption was that "there will be 5.0
> on summit with TCM and Accord and it somehow just happens". More
> transparent communication where we are at with high-profile CEPs like these
> and knowing if deadlines are going to be met would be welcome.
>
> I don't want to be that guy and don't take me wrong here, but really,
> these CEPs are being developed, basically, by devs from two companies,
> which have developers who do not have any real need to explain themselves
> like what they do, regularly, to outsiders. (or maybe you do, you just
> don't have time?) I get that. But on the other hand, you can not
> realistically expect that other folks will have any visibility into what is
> going on there and that there is a delay on the horizon and so on.
>
> 
> From: Maxim Muzafarov 
> Sent: Thursday, October 26, 2023 12:21
> To: dev@cassandra.apache.org
> Subject: Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an
> immediate 5.1-alpha1)
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
>
> Personally, I think frequent releases (2-3 per year) are better than
> infrequent big releases. I can understand all the concerns from a
> marketing perspective, as smaller major releases may not shine as
> brightly as a single "game changer" release. However, smaller
> releases, especially if they don't have backwards compatibility
> issues, are better for the engineering and SRE teams because if a
> long-awaited feature is delayed for any reason, there should be no
> worry about getting it in right into the next release.
>
> An analogy here might be that if you miss your train (small release)
> due to circumstances, you can wait right here for the next one, but if
> you miss a flight (big release), you will go back home :-) This is why
> I think that the 5.0, 5.1, 5.2, etc. are better and I support Mick's
> plan with the caveat that we should release 5.1 when 

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Benjamin Lerer]

 without CI being green, and merges without the
> CI being completely green are from the fact that our trunk CI has rarely
> been completely green, so we allow merging things which do not introduce
> NEW regressions, and we allow releases with known regressions that are
> deemed acceptable.
>
> We can indeed always vote to override it, and if it comes to that we can
> consider that as an option.
>
> -Jeremiah
>
> On Oct 31, 2023 at 11:41:29 AM, Benedict  wrote:
>
>
>
> That vote thread also did not reach the threshold; it was incorrectly
> counted, as committer votes are not binding for procedural changes. I
> counted at most 8 PMC +1 votes.
>
> The focus of that thread was also clearly GA releases and merges on such
> branches, since there was a focus on releases being failure-free. But this
> predates the more general release lifecycle vote that allows for alphas to
> have failing tests - which logically would be impossible if nothing were
> merged with failing or flaky tests.
>
> Either way, the vote and discussion specifically allow for this to be
> overridden.
>
> 🤷‍♀️
>
>
> On 31 Oct 2023, at 16:29, Jeremiah Jordan 
> wrote:
>
> 
> I never said there was a need for green CI for alpha.  We do have a
> requirement for not merging things to trunk that have known regressions in
> CI.
> Vote here:
> https://lists.apache.org/thread/j34mrgcy9wrtn04nwwymgm6893h0xwo9
>
>
>
> On Oct 31, 2023 at 3:23:48 AM, Benedict  wrote:
>
>
>
> There is no requirement for green CI on alpha. We voted last year to
> require running all tests before commit and to require green CI for beta
> releases. This vote was invalid because it didn’t reach the vote floor for
> a procedural change but anyway is not inconsistent with knowingly and
> selectively merging work without green CI.
>
> If we reach the summit we should take a look at the state of the PRs and
> make a decision about if they are alpha quality; if so, and we want a
> release, we should simply merge it and release. Making up a new release
> type when the work meets alpha standard to avoid an arbitrary and not
> mandated commit bar seems the definition of silly.
>
>
> On 31 Oct 2023, at 04:34, J. D. Jordan  wrote:
>
> 
>
> That is my understanding as well. If the TCM and Accord based on TCM
> branches are ready to commit by ~12/1 we can cut a 5.1 branch and then a
> 5.1-alpha release.
> Where “ready to commit” means our usual things of two committer +1 and
> green CI etc.
>
> If we are not ready to commit then I propose that as long as everything in
> the accord+tcm Apache repo branch has had two committer +1’s, but maybe
> people are still working on fixes for getting CI green or similar, we cut a
> 5.1-preview  build from the feature branch to vote on with known issues
> documented.  This would not be the preferred path, but would be a way to
> have a voted on release for summit.
>
> -Jeremiah
>
> On Oct 30, 2023, at 5:59 PM, Mick Semb Wever  wrote:
>
> 
>
> Hoping we can get clarity on this.
>
> The proposal was, once TCM and Accord merges to trunk,  then immediately
> branch cassandra-5.1 and cut an immediate 5.1-alpha1 release.
>
> This was to focus on stabilising TCM and Accord as soon as it lands, hence
> the immediate branching.
>
> And the alpha release as that is what our Release Lifecycle states it to
> be.
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>
> My understanding is that there was no squeezing in extra features into 5.1
> after TCM+Accord lands, and there's no need for a "preview" release – we
> move straight to the alpha, as our lifecycle states.  And we will describe
> all usability shortcomings and bugs with the alpha, our lifecycle docs
> permit this, if we feel the need to.
>
> All this said, if TCM does not merge before the Summit, and we want to get
> a release into user hands, it has been suggested we cut a preview release
> 5.1-preview1 off the feature branch.  This is a different scenario, and
> only a mitigation plan.
>
>
>
>
> On Thu, 26 Oct 2023 at 14:20, Benedict  wrote:
>
>
> The time to stabilise is orthogonal to the time we branch. Once we branch
> we stop accepting new features for the branch, and work to stabilise.
>
> My understanding is we will branch as soon as we have a viable alpha
> containing TCM and Accord. That means pretty soon after they land in the
> project, which we expect to be around the summit.
>
> If this isn’t the expectation we should make that clear, as it will affect
> how this decision is made.
>
>
> On 26 Oct 2023, at 10:14, Benjamin Lerer  wrote:
>
> 
>
> Regarding the release of 5.1

Re: Road to 5.0-GA (was: [VOTE] Release Apache Cassandra 5.0-alpha2)

[Benjamin Lerer]

I created a Dashboard to track the progress and remaining tasks for 5.0:
https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=593
Everybody logged in should have access. Ping me if it is not the case.

Le sam. 4 nov. 2023 à 19:54, Mick Semb Wever  a écrit :

>
> Please mark such bugs with fixVersion 5.0-beta
>
> If there are no more tickets that need API changes (i.e. those that should
> be marked fixVersion 5.0-alpha) this then indicates we do not need a
> 5.0-alpha3 release and can focus towards 5.0-beta1 (regardless of having
> blockers open to it).
>
> Appreciate the attention 18993 is getting – we do have a shortlist of
> beta blockers that we gotta prioritise !
>
>
> On Sat, 4 Nov 2023 at 18:33, Benedict  wrote:
>
>> Yep, data loss bugs are not any old bug. I’m concretely -1 (binding)
>> releasing a beta with one that’s either under investigation or confirmed.
>>
>> As Scott says, hopefully it won’t come to that - the joy of deterministic
>> testing is this should be straightforward to triage.
>>
>> On 4 Nov 2023, at 17:30, C. Scott Andreas  wrote:
>>
>> I’d happily be the first to vote -1(nb) on a release containing a known
>> and reproducible bug that can result in data loss or an incorrect response
>> to a query. And I certainly wouldn’t run it.
>>
>> Since we have a programmatic repro within just a few seconds, this should
>> not take long to root-cause.
>>
>> On Friday, Alex worked to get this reproducing on a Cassandra branch
>> rather than via unstaged changes. We should have a published / shareable
>> example with details near the beginning of the week.
>>
>> – Scott
>>
>> On Nov 4, 2023, at 10:17 AM, Josh McKenzie  wrote:
>>
>> 
>>
>> I think before we cut a beta we need to have diagnosed and fixed 18993
>> (assuming it is a bug).
>>
>> Before a beta? I could see that for rc or GA definitely, but having a
>> known (especially non-regressive) data loss bug in a beta seems like it's
>> compatible with the guarantees we're providing for it:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle
>>
>> This release is recommended for test/QA clusters where short(order of
>> minutes) downtime during upgrades is not an issue
>>
>>
>> On Sat, Nov 4, 2023, at 12:56 PM, Ekaterina Dimitrova wrote:
>>
>> Totally agree with the others. Such an issue on its own should be a
>> priority in any release. Looking forward to the reproduction test mentioned
>> on the ticket.
>>
>> Thanks to Alex for his work on harry!
>>
>> On Sat, 4 Nov 2023 at 12:47, Benedict  wrote:
>>
>> Alex can confirm but I think it actually turns out to be a new bug in
>> 5.0, but either way we should not cut a release with such a serious
>> potential known issue.
>>
>> > On 4 Nov 2023, at 16:18, J. D. Jordan 
>> wrote:
>> >
>> > Sounds like 18993 is not a regression in 5.0? But present in 4.1 as
>> well?  So I would say we should fix it with the highest priority and get a
>> new 4.1.x released. Blocking 5.0 beta voting is a secondary issue to me if
>> we have a “data not being returned” issue in an existing release?
>> >
>> >> On Nov 4, 2023, at 11:09 AM, Benedict  wrote:
>> >>
>> >> I think before we cut a beta we need to have diagnosed and fixed
>> 18993 (assuming it is a bug).
>> >>
>>  On 4 Nov 2023, at 16:04, Mick Semb Wever  wrote:
>> >>>
>> >>> 
>> 
>>  With the publication of this release I would like to switch the
>>  default 'latest' docs on the website from 4.1 to 5.0.  Are there any
>>  objections to this ?
>> >>>
>> >>>
>> >>> I would also like to propose the next 5.0 release to be 5.0-beta1
>> >>>
>> >>> With the aim of reaching GA for the Summit, I would like to suggest we
>> >>> work towards the best-case scenario of 5.0-beta1 in two weeks and
>> >>> 5.0-rc1 first week Dec.
>> >>>
>> >>> I know this is a huge ask with lots of unknowns we can't actually
>> >>> commit to.  But I believe it is a worthy goal, and possible if nothing
>> >>> sideswipes us – but we'll need all the help we can get this month to
>> >>> make it happen.
>> >>
>>
>>
>>

Re: Road to 5.0-GA (was: [VOTE] Release Apache Cassandra 5.0-alpha2)

[Benjamin Lerer]

Sorry for that. It should be fixed for everybody now.

Le lun. 6 nov. 2023 à 11:43, Miklosovic, Stefan via dev <
dev@cassandra.apache.org> a écrit :

> I can't view it either.
>
> 
> From: guo Maxwell 
> Sent: Monday, November 6, 2023 11:40
> To: dev@cassandra.apache.org
> Subject: Re: Road to 5.0-GA (was: [VOTE] Release Apache Cassandra
> 5.0-alpha2)
>
> NetApp Security WARNING: This is an external email. Do not click links or
> open attachments unless you recognize the sender and know the content is
> safe.
>
>
>
> Do I need permission to view this link? When I open it, an error appears,
> saying “It may have been deleted or you don't have permission to view it.”
>
> Benjamin Lerer mailto:b.le...@gmail.com>>
> 于2023年11月6日周一 18:34写道：
> I created a Dashboard to track the progress and remaining tasks for 5.0:
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=593<
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FRapidBoard.jspa%3FrapidView%3D593&data=05%7C01%7CStefan.Miklosovic%40netapp.com%7C83db318dc59d4ace3ded08dbdeb4d68b%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638348640501244920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=aDmFrtaDdB0F4kEG%2BHbBiF52VHTvrEdIwL2RUQXX%2FbY%3D&reserved=0
> >
> Everybody logged in should have access. Ping me if it is not the case.
>
> Le sam. 4 nov. 2023 à 19:54, Mick Semb Wever  m...@apache.org>> a écrit :
>
> Please mark such bugs with fixVersion 5.0-beta
>
> If there are no more tickets that need API changes (i.e. those that should
> be marked fixVersion 5.0-alpha) this then indicates we do not need a
> 5.0-alpha3 release and can focus towards 5.0-beta1 (regardless of having
> blockers open to it).
>
> Appreciate the attention 18993 is getting – we do have a shortlist of beta
> blockers that we gotta prioritise !
>
>
> On Sat, 4 Nov 2023 at 18:33, Benedict  bened...@apache.org>> wrote:
> Yep, data loss bugs are not any old bug. I’m concretely -1 (binding)
> releasing a beta with one that’s either under investigation or confirmed.
>
> As Scott says, hopefully it won’t come to that - the joy of deterministic
> testing is this should be straightforward to triage.
>
> On 4 Nov 2023, at 17:30, C. Scott Andreas  sc...@paradoxica.net>> wrote:
>
> I’d happily be the first to vote -1(nb) on a release containing a known
> and reproducible bug that can result in data loss or an incorrect response
> to a query. And I certainly wouldn’t run it.
>
> Since we have a programmatic repro within just a few seconds, this should
> not take long to root-cause.
>
> On Friday, Alex worked to get this reproducing on a Cassandra branch
> rather than via unstaged changes. We should have a published / shareable
> example with details near the beginning of the week.
>
> – Scott
>
> On Nov 4, 2023, at 10:17 AM, Josh McKenzie  jmcken...@apache.org>> wrote:
>
> 
> I think before we cut a beta we need to have diagnosed and fixed 18993
> (assuming it is a bug).
> Before a beta? I could see that for rc or GA definitely, but having a
> known (especially non-regressive) data loss bug in a beta seems like it's
> compatible with the guarantees we're providing for it:
> https://cwiki.apache.org/confluence/display/CASSANDRA/Release+Lifecycle<
> https://nam04.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCASSANDRA%2FRelease%2BLifecycle&data=05%7C01%7CStefan.Miklosovic%40netapp.com%7C83db318dc59d4ace3ded08dbdeb4d68b%7C4b0911a0929b4715944bc03745165b3a%7C0%7C0%7C638348640501244920%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C&sdata=lfB59qRc64YbPS9vGECYUYm4j2YHtwMQNe%2FiqafSQTk%3D&reserved=0
> >
>
> This release is recommended for test/QA clusters where short(order of
> minutes) downtime during upgrades is not an issue
>
>
> On Sat, Nov 4, 2023, at 12:56 PM, Ekaterina Dimitrova wrote:
> Totally agree with the others. Such an issue on its own should be a
> priority in any release. Looking forward to the reproduction test mentioned
> on the ticket.
>
> Thanks to Alex for his work on harry!
>
> On Sat, 4 Nov 2023 at 12:47, Benedict  bened...@apache.org>> wrote:
> Alex can confirm but I think it actually turns out to be a new bug in 5.0,
> but either way we should not cut a release with such a serious potential
> known issue.
>
> > On 4 Nov 2023, at 16:18, J. D. Jordan  jeremiah.jor...@gmail.com>> wrote:
> >
> > Sounds like 18993 is not a regres

Re: CEP-21 - Transactional cluster metadata merged to trunk

[Benjamin Lerer]

Hi,

I must admit that I have been surprised by this merge and this following
email. We had lengthy discussions recently and the final agreement was that
the requirement for a merge was a green CI.
I could understand that for some reasons as a community we could wish to
make some exceptions. In this present case there was no official discussion
to ask for an exception.
I believe that this merge creates a bad precedent where anybody can feel
entitled to merge without a green CI and disregard any previous community
agreement.

Le sam. 25 nov. 2023 à 09:22, Mick Semb Wever  a écrit :

>
> Great work Sam, Alex & Marcus !
>
>
>
>> There are about 15-20 flaky or failing tests in total, spread over
>> several test jobs[2] (i.e. single digit failures in a few of these). We
>> have filed JIRAs for the failures and are working on getting those fixed as
>> a top priority. CASSANDRA-19055[3] is the umbrella ticket for this follow
>> up work.
>>
>> There are also a number of improvements we will work on in the coming
>> weeks, we will file JIRAs for those early next week and add them as
>> subtasks to CASSANDRA-19055.
>>
>
>
> Can we get these tests temporarily annotated as skipped while all the
> subtickets to 19055 are being worked on ?
>
> As we have seen from CASSANDRA-18166 and CASSANDRA-19034 there's a lot of
> overhead now on 5.0 tickets having to navigate around these failures in
> trunk CI runs.
>
> Also, we're still trying to figure out how to do repeated runs for a patch
> so big… (the list of touched tests was too long for circleci, i need to
> figure out what the limit is and chunk it into separate circleci configs) …
> and it probably makes sense to wait until most of 19055 is done (or tests
> are temporarily annotated as skipped).
>
>
>

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Benjamin Lerer]

Looking at the board it is unclear to me why CASSANDRA-19011
, CASSANDRA-19018
,  CASSANDRA-18796
 and  CASSANDRA-18940
 are not beta
tickets.
SAI being one of the important features of 5.0 it seems to me that those
tickets should have been handled for the beta release.
CASSANDRA-19039 
could also be a real problem.


Le dim. 26 nov. 2023 à 13:35, Mick Semb Wever  a écrit :

>
> Proposing the test build of Cassandra 5.0-beta1 for release.
>
> sha1: e0c0c31c7f6db1e3ddb80cef842b820fc27fd0eb
> Git: https://github.com/apache/cassandra/tree/5.0-beta1-tentative
> Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1319/org/apache/cassandra/cassandra-all/5.0-beta1/
>
> The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/5.0-beta1/
>
> The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
>
> Remaining tickets to get us to 5.0-rc1 can be found on this jira board:
> https://issues.apache.org/jira/secure/RapidBoard.jspa?rapidView=593&view=detail
>
> [1]: CHANGES.txt:
> https://github.com/apache/cassandra/blob/5.0-beta1-tentative/CHANGES.txt
> [2]: NEWS.txt:
> https://github.com/apache/cassandra/blob/5.0-beta1-tentative/NEWS.txt
>

Re: [DISCUSS] Harry in-tree

[Benjamin Lerer]

+1

Le lun. 27 nov. 2023 à 18:01, Brandon Williams  a écrit :

> I am +1 on including Harry in-tree.
>
> Kind Regards,
> Brandon
>
> On Fri, Nov 24, 2023 at 9:44 AM Alex Petrov  wrote:
> >
> > Hi everyone,
> >
> > With TCM landed, there will be way more Harry tests in-tree: we are
> using it for many coordination tests, and there's now a simulator test that
> uses Harry. During development, Harry has allowed us to uncover and resolve
> numerous elusive edge cases.
> >
> > I had conversations with several folks, and wanted to propose to move
> harry-core to Cassandra test tree. This will substantially
> simplify/streamline co-development of Cassandra and Harry. With a new
> HistoryBuilder API that has helped to find and trigger [1] [2] and [3], it
> will also be much more approachable.
> >
> > Besides making it easier for everyone to develop new fuzz tests, it will
> also substantially lower the barrier to entry. Currently, debugging an
> issue found by Harry involves a cumbersome process of rebuilding and
> transferring jars between Cassandra and Harry, depending on which side you
> modify. This not only hampers efficiency but also deters broader adoption.
> By merging harry-core into the Cassandra test tree, we eliminate this
> barrier.
> >
> > Thank you,
> > --Alex
> >
> > [1] https://issues.apache.org/jira/browse/CASSANDRA-19011
> > [2] https://issues.apache.org/jira/browse/CASSANDRA-18993
> > [3] https://issues.apache.org/jira/browse/CASSANDRA-18932
>

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Benjamin Lerer]

-1 based on the problems raised by Caleb.

I would be fine with releasing that version as an alpha as Jeremiah
proposed.

As of this time, I'm also not aware of a user of the project operating a
> build from the 5.0 branch at substantial scale to suss out the operational
> side of what can be expected. If someone is running a build supporting
> non-perf-test traffic derived from the 5.0 branch and has an experience
> report to share it would be great to read.


Some people at Datastax are working on such testing. It will take a bit of
time before we get the final results though.

Le mar. 28 nov. 2023 à 19:27, J. D. Jordan  a
écrit :

> That said. This is clearly better than and with many fixes from the alpha.
> Would people be more comfortable if this cut was released as another alpha
> and we do beta1 once the known fixes land?
>
> On Nov 28, 2023, at 12:21 PM, J. D. Jordan 
> wrote:
>
> 
> -0 (NB) on this cut. Given the concerns expressed so far in the thread I
> would think we should re-cut beta1 at the end of the week.
>
> On Nov 28, 2023, at 12:06 PM, Patrick McFadin  wrote:
>
> 
> I'm a +1 on a beta now vs maybe later. Beta doesn't imply perfect
> especially if there are declared known issues. We need people outside of
> this tight group using it and finding issues. I know how this rolls. Very
> few people touch a Alpha release. Beta is when the engine starts and we
> need to get it started asap. Otherwise we are telling ourselves we have the
> perfect testing apparatus and don't need more users testing. I don't think
> that is the case.
>
> Scott, Ekaterina, and I are going to be on stage in 2 weeks talking about
> Cassandra 5 in the keynotes. In that time, our call to action is going to
> be to test the beta.
>
> Patrick
>
> On Tue, Nov 28, 2023 at 9:41 AM Mick Semb Wever  wrote:
>
>> The vote will be open for 72 hours (longer if needed). Everyone who has
>>> tested the build is invited to vote. Votes by PMC members are considered
>>> binding. A vote passes if there are at least three binding +1s and no -1's.
>>>
>>
>>
>> +1
>>
>> Checked
>> - signing correct
>> - checksums are correct
>> - source artefact builds (JDK 11+17)
>> - binary artefact runs (JDK 11+17)
>> - debian package runs (JDK 11+17)
>> - debian repo runs (JDK 11+17)
>> - redhat* package runs (JDK11+17)
>> - redhat* repo runs (JDK 11+17)
>>
>>
>> With the disclaimer:  There's a few known bugs in SAI, e.g. 19011, with
>> fixes to be available soon in 5.0-beta2.
>>
>>
>>

Re: Welcome Francisco Guerrero Hernandez as Cassandra Committer

[Benjamin Lerer]

Congratulations!!! Well deserved!

Le mer. 29 nov. 2023 à 07:31, Berenguer Blasi  a
écrit :

> Welcome!
> On 29/11/23 2:24, guo Maxwell wrote:
>
> Congrats!
>
> Jacek Lewandowski  于2023年11月29日周三 06:16写道：
>
>> Congrats!!!
>>
>> wt., 28 lis 2023, 23:08 użytkownik Abe Ratnofsky  napisał:
>>
>>> Congrats Francisco!
>>>
>>> > On Nov 28, 2023, at 1:56 PM, C. Scott Andreas 
>>> wrote:
>>> >
>>> > Congratulations, Francisco!
>>> >
>>> > - Scott
>>> >
>>> >> On Nov 28, 2023, at 10:53 AM, Dinesh Joshi  wrote:
>>> >>
>>> >> The PMC members are pleased to announce that Francisco Guerrero
>>> Hernandez has accepted
>>> >> the invitation to become committer today.
>>> >>
>>> >> Congratulations and welcome!
>>> >>
>>> >> The Apache Cassandra PMC members
>>>
>>>

Re: [VOTE] Release Apache Cassandra 5.0-beta1 (take2)

[Benjamin Lerer]

+1

Le mar. 5 déc. 2023 à 12:23, Maxim Muzafarov  a écrit :

> +1 (nb)
>
> run locally, executed some queries over vts
>
> On Mon, 4 Dec 2023 at 15:15, Brandon Williams  wrote:
> >
> > +1
> >
> > Kind Regards,
> > Brandon
> >
> > On Fri, Dec 1, 2023 at 7:32 AM Mick Semb Wever  wrote:
> > >
> > >
> > > Proposing the test build of Cassandra 5.0-beta1 for release.
> > >
> > > sha1: 87fd1fa88a0c859cc32d9f569ad09ad0b345e465
> > > Git: https://github.com/apache/cassandra/tree/5.0-beta1-tentative
> > > Maven Artifacts:
> https://repository.apache.org/content/repositories/orgapachecassandra-1321/org/apache/cassandra/cassandra-all/5.0-beta1/
> > >
> > > The Source and Build Artifacts, and the Debian and RPM packages and
> repositories, are available here:
> https://dist.apache.org/repos/dist/dev/cassandra/5.0-beta1/
> > >
> > > The vote will be open for 72 hours (longer if needed). Everyone who
> has tested the build is invited to vote. Votes by PMC members are
> considered binding. A vote passes if there are at least three binding +1s
> and no -1's.
> > >
> > > [1]: CHANGES.txt:
> https://github.com/apache/cassandra/blob/5.0-beta1-tentative/CHANGES.txt
> > > [2]: NEWS.txt:
> https://github.com/apache/cassandra/blob/5.0-beta1-tentative/NEWS.txt
>

Welcome Mike Adamson as Cassandra committer

[Benjamin Lerer]

 The PMC members are pleased to announce that Mike Adamson has accepted
the invitation to become committer.

Thanks a lot, Mike, for everything you have done for the project.

Congratulations and welcome

The Apache Cassandra PMC members

Re: [VOTE] Release Apache Cassandra Java Driver 4.18.0

[Benjamin Lerer]

+1

Le lun. 11 déc. 2023 à 19:54, Brandon Williams  a écrit :

> +1
>
> Kind Regards,
> Brandon
>
> On Sat, Dec 9, 2023 at 1:43 AM Mick Semb Wever  wrote:
> >
> > Proposing the test build of Cassandra Java Driver 4.18.0 for release.
> >
> > sha1: 105d378fce16804a8af4c26cf732340a0c63b3c9
> > Git: https://github.com/apache/cassandra-java-driver/tree/4.18.0
> > Maven Artifacts:
> >
> https://repository.apache.org/content/repositories/orgapachecassandra-1322
> >
> > The Source release and Binary convenience artifacts are available here:
> >
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-java-driver/4.18.0/
> >
> > This is the first release post-donation of the Java Driver.  The maven
> coordinates have changed from com.datastax.oss to org.apache.cassandra,
> while all package names remain the same.  There is still work to be done on
> a number of fronts, e.g. being vendor-neutrality, covered under
> CASSANDRA-18611.
> >
> > The vote will be open for 72 hours (longer if needed). Everyone who has
> tested the build is invited to vote. Votes by PMC members are considered
> binding. A vote passes if there are at least three binding +1s and no -1's.
> >
> >
>

[DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

Hi everybody,

I would like to open the discussion on the introduction of a cost based
optimizer to allow Cassandra to pick the best execution plan based on the
data distribution.Therefore, improving the overall query performance.

This CEP should also lay the groundwork for the future addition of features
like joins, subqueries, OR/NOT and index ordering.

The proposal is here:
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-39%3A+Cost+Based+Optimizer

Thank you in advance for your feedback.

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

g guarantees). Of
> course, re-preparing a query might lead to a new plan, though any
> coordinators with the query in their cache should be able to retrieve it
> cheaply. If the execution model is efficiently serialised this might have
> the ancillary benefit of improving the occupancy of our prepared query
> cache.
>
> On 13 Dec 2023, at 00:44, Jon Haddad  wrote:
>
> 
> I think it makes sense to see what the actual overhead is of CBO before
> making the assumption it'll be so high that we need to have two code
> paths.  I'm happy to provide thorough benchmarking and analysis when it
> reaches a testing phase.
>
> I'm excited to see where this goes.  I think it sounds very forward
> looking and opens up a lot of possibilities.
>
> Jon
>
> On Tue, Dec 12, 2023 at 4:25 PM guo Maxwell  wrote:
>
>> Nothing expresses my thoughts better than +1
>> ，It feels like it means a lot to Cassandra.
>>
>> I have a question. Is it easy to turn off cbo's optimizer or by pass in
>> some way? Because some simple read and write requests will have better
>> performance without cbo, which is also the advantage of Cassandra compared
>> to some rdbms.
>>
>>
>> David Capwell 于2023年12月13日 周三上午3:37写道：
>>
>>> Overall LGTM.
>>>
>>>
>>> On Dec 12, 2023, at 5:29 AM, Benjamin Lerer  wrote:
>>>
>>> Hi everybody,
>>>
>>> I would like to open the discussion on the introduction of a cost based
>>> optimizer to allow Cassandra to pick the best execution plan based on the
>>> data distribution.Therefore, improving the overall query performance.
>>>
>>> This CEP should also lay the groundwork for the future addition of
>>> features like joins, subqueries, OR/NOT and index ordering.
>>>
>>> The proposal is here:
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-39%3A+Cost+Based+Optimizer
>>>
>>> Thank you in advance for your feedback.
>>>
>>>
>>>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

>
> Can you share the reasons why Apache Calcite is not suitable for this case
> and why it was rejected


My understanding is that Calcite was made for two main things: to help with
optimizing SQL-like languages and to let people query different kinds of
data sources together.

We could think about using it for our needs, but there are some big
problems:

   1.

   CQL is not SQL. There are significant differences between the 2 languages
   2.

   Cassandra has its own specificities that will influence the cost model
   and the way we deal with optimizations: partitions, replication factors,
   consistency levels, LSM tree storage, ...
   3.

   Every framework comes with its own limitations and additional cost

>From my view, there are too many big differences between what Calcite does
and what we need in Cassandra. If we used Calcite, it would also mean
relying a lot on another system that everyone would have to learn and
adjust to. The problems and extra work this would bring don't seem worth
the benefits we might get


Le mer. 13 déc. 2023 à 18:06, Benjamin Lerer  a écrit :

> One thing that I did not mention is the fact that this CEP is only a high
> level proposal. There will be deeper discussions on the dev list around the
> different parts of this proposal when we reach those parts and have enough
> details to make those discussions more meaningful.
>
>
>> The maintenance and distribution of summary statistics in particular is
>> worthy of its own CEP, and it might be preferable to split it out.
>
>
> For maintaining node statistics the idea is to re-use the current
> Memtable/SSTable mechanism and relies on mergeable statistics. That will
> allow us to easily build node level statistics for a given table by merging
> all the statistics of its memtable and SSTables. For the distribution of
> these node statistics we are still exploring different options. We can come
> back with a precise proposal once we have hammered all the details.
> Is it for you a blocker for this CEP or do you just want to make sure that
> this part is discussed in deeper details before we implement it?
>
>>
>> The proposal also seems to imply we are aiming for coordinators to all
>> make the same decision for a query, which I think is challenging, and it
>> would be worth fleshing out the design here a little (perhaps just in Jira).
>
>
> The goal is that the large majority of nodes preparing a query at a given
> point in time should make the same decision and that over time all nodes
> should converge toward the same decision. This part is dependent on the
> node statistics distribution, the cost model and the triggers for
> re-optimization (that will require some experimentation).
>
> There’s also not much discussion of the execution model: I think it would
>> make most sense for this to be independent of any cost and optimiser models
>> (though they might want to operate on them), so that EXPLAIN and hints can
>> work across optimisers (a suitable hint might essentially bypass the
>> optimiser, if the optimiser permits it, by providing a standard execution
>> model)
>>
>
> It is not clear to me what you mean by "a standard execution model"?
> Otherwise, we were not planning to have the execution model or the hints
> depending on the optimizer.
>
> I think it would be worth considering providing the execution plan to the
>> client as part of query preparation, as an opaque payload to supply to
>> coordinators on first contact, as this might simplify the problem of
>> ensuring queries behave the same without adopting a lot of complexity for
>> synchronising statistics (which will never provide strong guarantees). Of
>> course, re-preparing a query might lead to a new plan, though any
>> coordinators with the query in their cache should be able to retrieve it
>> cheaply. If the execution model is efficiently serialised this might have
>> the ancillary benefit of improving the occupancy of our prepared query
>> cache.
>>
>
> I am not sure that I understand your proposal. If 2 nodes build a
> different execution plan how do you solve that conflict?
>
> Le mer. 13 déc. 2023 à 09:55, Benedict  a écrit :
>
>> A CBO can only make worse decisions than the status quo for what I
>> presume are the majority of queries - i.e. those that touch only primary
>> indexes. In general, there are plenty of use cases that prefer determinism.
>> So I agree that there should at least be a CBO implementation that makes
>> the same decisions as the status quo, deterministically.
>>
>>
>> I do support the proposal, but would like to see some elements discussed
>> in more detail. The maintenance and distribution of summar

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

>
> I mean that an important part of this work - not specified in the CEP
> (AFAICT) - should probably be to define some standard execution model, that
> we can manipulate and serialise, for use across (and without) optimisers.


I am confused because for me an execution model defines how operations are
executed within the database in a conceptual way, which is not something
that this CEP intends to change. Do you mean the physical/execution plan?
Today this plan is somehow represented for reads by the SelectStatement and
its components (Selections, StatementRestrictions, ...) it is then
converted at execution time after parameter binding into a ReadCommand
which is sent to the replicas.
We plan to refactor SelectStatement and its components but the ReadCommands
change should be relatively small. What you are proposing is not part of
the scope of this CEP.

Le jeu. 14 déc. 2023 à 10:24, Benjamin Lerer  a écrit :

> Can you share the reasons why Apache Calcite is not suitable for this case
>> and why it was rejected
>
>
> My understanding is that Calcite was made for two main things: to help
> with optimizing SQL-like languages and to let people query different kinds
> of data sources together.
>
> We could think about using it for our needs, but there are some big
> problems:
>
>1.
>
>CQL is not SQL. There are significant differences between the 2
>languages
>2.
>
>Cassandra has its own specificities that will influence the cost model
>and the way we deal with optimizations: partitions, replication factors,
>consistency levels, LSM tree storage, ...
>3.
>
>Every framework comes with its own limitations and additional cost
>
> From my view, there are too many big differences between what Calcite does
> and what we need in Cassandra. If we used Calcite, it would also mean
> relying a lot on another system that everyone would have to learn and
> adjust to. The problems and extra work this would bring don't seem worth
> the benefits we might get
>
>
> Le mer. 13 déc. 2023 à 18:06, Benjamin Lerer  a écrit :
>
>> One thing that I did not mention is the fact that this CEP is only a high
>> level proposal. There will be deeper discussions on the dev list around the
>> different parts of this proposal when we reach those parts and have enough
>> details to make those discussions more meaningful.
>>
>>
>>> The maintenance and distribution of summary statistics in particular is
>>> worthy of its own CEP, and it might be preferable to split it out.
>>
>>
>> For maintaining node statistics the idea is to re-use the current
>> Memtable/SSTable mechanism and relies on mergeable statistics. That will
>> allow us to easily build node level statistics for a given table by merging
>> all the statistics of its memtable and SSTables. For the distribution of
>> these node statistics we are still exploring different options. We can come
>> back with a precise proposal once we have hammered all the details.
>> Is it for you a blocker for this CEP or do you just want to make sure
>> that this part is discussed in deeper details before we implement it?
>>
>>>
>>> The proposal also seems to imply we are aiming for coordinators to all
>>> make the same decision for a query, which I think is challenging, and it
>>> would be worth fleshing out the design here a little (perhaps just in Jira).
>>
>>
>> The goal is that the large majority of nodes preparing a query at a given
>> point in time should make the same decision and that over time all nodes
>> should converge toward the same decision. This part is dependent on the
>> node statistics distribution, the cost model and the triggers for
>> re-optimization (that will require some experimentation).
>>
>> There’s also not much discussion of the execution model: I think it would
>>> make most sense for this to be independent of any cost and optimiser models
>>> (though they might want to operate on them), so that EXPLAIN and hints can
>>> work across optimisers (a suitable hint might essentially bypass the
>>> optimiser, if the optimiser permits it, by providing a standard execution
>>> model)
>>>
>>
>> It is not clear to me what you mean by "a standard execution model"?
>> Otherwise, we were not planning to have the execution model or the hints
>> depending on the optimizer.
>>
>> I think it would be worth considering providing the execution plan to the
>>> client as part of query preparation, as an opaque payload to supply to
>>> coordinators on first contact, as this might simplify the problem of
>>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

The binding of the parser output to the schema (what is today the
Raw.prepare call) will create the logical plan, expressed as a tree of
relational operators. Simplification and normalization will happen on that
tree to produce a new equivalent logical plan. That logical plan will be
used as input to the optimizer. The output will be a physical plan
producing the output specified by the logical plan. A tree of physical
operators specifying how the operations should be performed.

That physical plan will be stored as part of the statements
(SelectStatement, ModificationStatement, ...) in the prepared statement
cache. Upon execution, variables will be bound and the
RangeCommands/Mutations will be created based on the physical plan.

The string representation of a physical plan will effectively represent the
output of an EXPLAIN statement but outside of that the physical plan will
stay encapsulated within the statement classes.
Hints will be parameters provided to the optimizer to enforce some specific
choices. Like always using an Index Scan instead of a Table Scan, ignoring
the cost comparison.

So yes, this physical plan is the structure that you have in mind but the
idea of sharing it is not part of the CEP. I did not document it because it
will simply be a tree of physical operators used internally.

My proposal is that the execution plan of the coordinator that prepares a
> query gets serialised to the client, which then provides the execution plan
> to all future coordinators, and coordinators provide it to replicas as
> necessary.
>
>
> This means it is not possible for any conflict to arise for a single
> client. It would guarantee consistency of execution for any single client
> (and avoid any drift over the client’s sessions), without necessarily
> guaranteeing consistency for all clients.
>

 It seems that there is a difference between the goal of your proposal and
the one of the CEP. The goal of the CEP is first to ensure optimal
performance. It is ok to change the execution plan for one that delivers
better performance. What we want to minimize is having a node performing
queries in an inefficient way for a long period of time.

The client side proposal targets consistency for a given query on a given
driver instance. In practice, it would be possible to have 2 similar
queries with 2 different execution plans on the same driver making things
really confusing. Identifying the source of an inefficient query will also
be pretty hard.

Interestingly, having 2 nodes with 2 different execution plans might not be
a serious problem. It simply means that based on cardinality at t1, the
optimizer on node 1 chose plan 1 while the one on node 2 chose plan 2 at
t2. In practice if the cost estimates reflect properly the actual cost
those 2 plans should have pretty similar efficiency. The problem is more
about the fact that you would ideally want a uniform behavior around your
cluster.
Changes of execution plans should only occur at certain points. So the main
problematic scenario is when the data distribution is around one of those
points. Which is also the point where the change should have the least
impact.


Le jeu. 14 déc. 2023 à 11:38, Benedict  a écrit :

> There surely needs to be a more succinct and abstract representation in
> order to perform transformations on the query plan? You don’t intend to
> manipulate the object graph directly as you apply any transformations when
> performing simplification or cost based analysis? This would also (I
> expect) be the form used to support EXPLAIN functionality, and probably
> also HINTs etc. This would ideally *not* be coupled to the CBO itself,
> and would ideally be succinctly serialised.
>
> I would very much expect the query plan to be represented abstractly as
> part of this work, and for there to be a mechanism that translates this
> abstract representation into the object graph that executes it.
>
> If I’m incorrect, could you please elaborate more specifically how you
> intend to go about this?
>
> On 14 Dec 2023, at 10:33, Benjamin Lerer  wrote:
>
> 
>
>> I mean that an important part of this work - not specified in the CEP
>> (AFAICT) - should probably be to define some standard execution model, that
>> we can manipulate and serialise, for use across (and without) optimisers.
>
>
> I am confused because for me an execution model defines how operations are
> executed within the database in a conceptual way, which is not something
> that this CEP intends to change. Do you mean the physical/execution plan?
> Today this plan is somehow represented for reads by the SelectStatement
> and its components (Selections, StatementRestrictions, ...) it is then
> converted at execution time after parameter binding into a ReadCommand
> which is sent to the replicas.
> We plan to refactor SelectStatement and its components but 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

>
>   So yes, this physical plan is the structure that you have in mind but
> the idea of sharing it is not part of the CEP.


Sorry, Benedict, what I meant by sharing was sharing across the nodes. It
is an integral part of the optimizer API that the CEP talks about as it
represents its output. I did not realize that this part was not clear.

Le jeu. 14 déc. 2023 à 17:15, Benedict  a écrit :

> Fwiw Chris, I agree with your concerns, but I think the introduction of a
> CBO - done right - is in principle a good thing in its own right. It’s
> independent of the issues you mention, even if it might enable features
> that exacerbate them.
>
> It should also help enable secondary indexes work better, which is I think
> their main justification. Otherwise there isn’t really a good reason for it
> today. Though I personally anticipate ongoing issues around 2i that SAI is
> perhaps over exuberantly sold as solving, and that a CBO will not fix. But
> we’ll see how that evolves.
>
>
>
> On 14 Dec 2023, at 15:49, Chris Lohfink  wrote:
>
> 
> I don't wanna be a blocker for this CEP or anything but did want to put my
> 2 cents in. This CEP is horrifying to me.
>
> I have seen thousands of clusters across multiple companies and helped
> them get working successfully. A vast majority of that involved blocking
> the use of MVs, GROUP BY, secondary indexes, and even just simple _range
> queries_. The "unncessary restrictions of cql" are not only necessary IMHO,
> more restrictions are necessary to be successful at scale. The idea of just
> opening up CQL to general purpose relational queries and lines like 
> "supporting
> queries with joins in an efficient way" ... I would really like us to
> make secondary indexes be a viable option before we start opening up
> floodgates on stuff like this.
>
> Chris
>
> On Thu, Dec 14, 2023 at 9:37 AM Benedict  wrote:
>
>> > So yes, this physical plan is the structure that you have in mind but
>> the idea of sharing it is not part of the CEP.
>>
>>
>> I think it should be. This should form a major part of the API on which
>> any CBO is built.
>>
>>
>> > It seems that there is a difference between the goal of your proposal
>> and the one of the CEP. The goal of the CEP is first to ensure optimal
>> performance. It is ok to change the execution plan for one that delivers
>> better performance. What we want to minimize is having a node performing
>> queries in an inefficient way for a long period of time.
>>
>>
>> You have made a goal of the CEP synchronising summary statistics across
>> the whole cluster in order to achieve some degree of uniformity of query
>> plan. So this is explicitly a goal of the CEP, and synchronising summary
>> statistics is a hard problem and won’t provide strong guarantees.
>>
>>
>> > The client side proposal targets consistency for a given query on a
>> given driver instance. In practice, it would be possible to have 2 similar
>> queries with 2 different execution plans on the same driver
>>
>>
>> This would only be possible if the driver permitted it. A driver could
>> (and should) enforce that it only permits one query plan per query.
>>
>>
>> The opposite is true for your proposal: some queries may begin degrading
>> because they touch specific replicas that optimise the query differently,
>> and this will be hard to debug.
>>
>>
>> On 14 Dec 2023, at 15:30, Benjamin Lerer  wrote:
>>
>> 
>> The binding of the parser output to the schema (what is today the
>> Raw.prepare call) will create the logical plan, expressed as a tree of
>> relational operators. Simplification and normalization will happen on that
>> tree to produce a new equivalent logical plan. That logical plan will be
>> used as input to the optimizer. The output will be a physical plan
>> producing the output specified by the logical plan. A tree of physical
>> operators specifying how the operations should be performed.
>>
>> That physical plan will be stored as part of the statements
>> (SelectStatement, ModificationStatement, ...) in the prepared statement
>> cache. Upon execution, variables will be bound and the
>> RangeCommands/Mutations will be created based on the physical plan.
>>
>> The string representation of a physical plan will effectively represent
>> the output of an EXPLAIN statement but outside of that the physical plan
>> will stay encapsulated within the statement classes.
>> Hints will be parameters provided to the optimizer to enforce some
>> specific choices. Like always using an Index Scan instead 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

posal targets consistency for a given query on a
>> given driver instance. In practice, it would be possible to have 2 similar
>> queries with 2 different execution plans on the same driver
>>
>>
>> This would only be possible if the driver permitted it. A driver could
>> (and should) enforce that it only permits one query plan per query.
>>
>>
>> The opposite is true for your proposal: some queries may begin degrading
>> because they touch specific replicas that optimise the query differently,
>> and this will be hard to debug.
>>
>>
>> On 14 Dec 2023, at 15:30, Benjamin Lerer  wrote:
>>
>> 
>> The binding of the parser output to the schema (what is today the
>> Raw.prepare call) will create the logical plan, expressed as a tree of
>> relational operators. Simplification and normalization will happen on that
>> tree to produce a new equivalent logical plan. That logical plan will be
>> used as input to the optimizer. The output will be a physical plan
>> producing the output specified by the logical plan. A tree of physical
>> operators specifying how the operations should be performed.
>>
>> That physical plan will be stored as part of the statements
>> (SelectStatement, ModificationStatement, ...) in the prepared statement
>> cache. Upon execution, variables will be bound and the
>> RangeCommands/Mutations will be created based on the physical plan.
>>
>> The string representation of a physical plan will effectively represent
>> the output of an EXPLAIN statement but outside of that the physical plan
>> will stay encapsulated within the statement classes.
>> Hints will be parameters provided to the optimizer to enforce some
>> specific choices. Like always using an Index Scan instead of a Table Scan,
>> ignoring the cost comparison.
>>
>> So yes, this physical plan is the structure that you have in mind but the
>> idea of sharing it is not part of the CEP. I did not document it because it
>> will simply be a tree of physical operators used internally.
>>
>> My proposal is that the execution plan of the coordinator that prepares a
>>> query gets serialised to the client, which then provides the execution plan
>>> to all future coordinators, and coordinators provide it to replicas as
>>> necessary.
>>>
>>>
>>> This means it is not possible for any conflict to arise for a single
>>> client. It would guarantee consistency of execution for any single client
>>> (and avoid any drift over the client’s sessions), without necessarily
>>> guaranteeing consistency for all clients.
>>>
>>
>>  It seems that there is a difference between the goal of your proposal
>> and the one of the CEP. The goal of the CEP is first to ensure optimal
>> performance. It is ok to change the execution plan for one that delivers
>> better performance. What we want to minimize is having a node performing
>> queries in an inefficient way for a long period of time.
>>
>> The client side proposal targets consistency for a given query on a given
>> driver instance. In practice, it would be possible to have 2 similar
>> queries with 2 different execution plans on the same driver making things
>> really confusing. Identifying the source of an inefficient query will also
>> be pretty hard.
>>
>> Interestingly, having 2 nodes with 2 different execution plans might not
>> be a serious problem. It simply means that based on cardinality at t1, the
>> optimizer on node 1 chose plan 1 while the one on node 2 chose plan 2 at
>> t2. In practice if the cost estimates reflect properly the actual cost
>> those 2 plans should have pretty similar efficiency. The problem is more
>> about the fact that you would ideally want a uniform behavior around your
>> cluster.
>> Changes of execution plans should only occur at certain points. So the
>> main problematic scenario is when the data distribution is around one of
>> those points. Which is also the point where the change should have the
>> least impact.
>>
>>
>> Le jeu. 14 déc. 2023 à 11:38, Benedict  a écrit :
>>
>>> There surely needs to be a more succinct and abstract representation in
>>> order to perform transformations on the query plan? You don’t intend to
>>> manipulate the object graph directly as you apply any transformations when
>>> performing simplification or cost based analysis? This would also (I
>>> expect) be the form used to support EXPLAIN functionality, and probably
>>> also HINTs etc. This would ideally *not* be coupled to the CBO itself,

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

HO,
>> more restrictions are necessary to be successful at scale. The idea of just
>> opening up CQL to general purpose relational queries and lines like 
>> "supporting
>> queries with joins in an efficient way" ... I would really like us to
>> make secondary indexes be a viable option before we start opening up
>> floodgates on stuff like this.
>>
>> Chris
>>
>> On Thu, Dec 14, 2023 at 9:37 AM Benedict  wrote:
>>
>>> > So yes, this physical plan is the structure that you have in mind but
>>> the idea of sharing it is not part of the CEP.
>>>
>>>
>>> I think it should be. This should form a major part of the API on which
>>> any CBO is built.
>>>
>>>
>>> > It seems that there is a difference between the goal of your proposal
>>> and the one of the CEP. The goal of the CEP is first to ensure optimal
>>> performance. It is ok to change the execution plan for one that delivers
>>> better performance. What we want to minimize is having a node performing
>>> queries in an inefficient way for a long period of time.
>>>
>>>
>>> You have made a goal of the CEP synchronising summary statistics across
>>> the whole cluster in order to achieve some degree of uniformity of query
>>> plan. So this is explicitly a goal of the CEP, and synchronising summary
>>> statistics is a hard problem and won’t provide strong guarantees.
>>>
>>>
>>> > The client side proposal targets consistency for a given query on a
>>> given driver instance. In practice, it would be possible to have 2 similar
>>> queries with 2 different execution plans on the same driver
>>>
>>>
>>> This would only be possible if the driver permitted it. A driver could
>>> (and should) enforce that it only permits one query plan per query.
>>>
>>>
>>> The opposite is true for your proposal: some queries may begin degrading
>>> because they touch specific replicas that optimise the query differently,
>>> and this will be hard to debug.
>>>
>>>
>>> On 14 Dec 2023, at 15:30, Benjamin Lerer  wrote:
>>>
>>> 
>>> The binding of the parser output to the schema (what is today the
>>> Raw.prepare call) will create the logical plan, expressed as a tree of
>>> relational operators. Simplification and normalization will happen on that
>>> tree to produce a new equivalent logical plan. That logical plan will be
>>> used as input to the optimizer. The output will be a physical plan
>>> producing the output specified by the logical plan. A tree of physical
>>> operators specifying how the operations should be performed.
>>>
>>> That physical plan will be stored as part of the statements
>>> (SelectStatement, ModificationStatement, ...) in the prepared statement
>>> cache. Upon execution, variables will be bound and the
>>> RangeCommands/Mutations will be created based on the physical plan.
>>>
>>> The string representation of a physical plan will effectively represent
>>> the output of an EXPLAIN statement but outside of that the physical plan
>>> will stay encapsulated within the statement classes.
>>> Hints will be parameters provided to the optimizer to enforce some
>>> specific choices. Like always using an Index Scan instead of a Table Scan,
>>> ignoring the cost comparison.
>>>
>>> So yes, this physical plan is the structure that you have in mind but
>>> the idea of sharing it is not part of the CEP. I did not document it
>>> because it will simply be a tree of physical operators used internally.
>>>
>>> My proposal is that the execution plan of the coordinator that prepares
>>>> a query gets serialised to the client, which then provides the execution
>>>> plan to all future coordinators, and coordinators provide it to replicas as
>>>> necessary.
>>>>
>>>>
>>>> This means it is not possible for any conflict to arise for a single
>>>> client. It would guarantee consistency of execution for any single client
>>>> (and avoid any drift over the client’s sessions), without necessarily
>>>> guaranteeing consistency for all clients.
>>>>
>>>
>>>  It seems that there is a difference between the goal of your proposal
>>> and the one of the CEP. The goal of the CEP is first to ensure optimal
>>> performance. It is ok to change the execution plan for one that delivers
>>> better per

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

tically give us a more gradual path to migration to a
> cost-based guardrail for instance, and would preserve the current
> robustness of the system while making it at least a touch more configurable.
>
> On Fri, Dec 15, 2023, at 11:03 AM, Chris Lohfink wrote:
>
> Thanks for time in addressing concerns. At least with initial versions, as
> long as there is a way to replace it with noop or disable it I would be
> happy. This is pretty standard practice with features nowadays but I wanted
> to highlight it as this might require some pretty tight coupling.
>
> Chris
>
> On Fri, Dec 15, 2023 at 7:57 AM Benjamin Lerer  wrote:
>
> Hey Chris,
> You raise some valid points.
>
> I believe that there are 3 points that you mentioned:
> 1) CQL restrictions are some form of safety net and should be kept
> 2) A lot of Cassandra features do not scale and/or are too easy to use in
> a wrong way that can make the whole system collapse. We should not add more
> to that list. Especially not joins.
>
> 3) Should we not start to fix features like secondary index rather than
> adding new ones? Which is heavily linked to 2).
>
> Feel free to correct me if I got them wrong or missed one.
>
> Regarding 1), I believe that you refer to the "Removing unnecessary CQL
> query limitations and inconsistencies" section. We are not planning to
> remove any safety net here.
> What we want to remove is a certain amount of limitations which make
> things confusing for a user trying to write a query for no good reason.
> Like "why can I define a column alias but not use it anywhere in my query?"
> or "Why can I not create a list with 2 bind parameters?". While refactoring
> some CQL code, I kept on finding those types of exceptions that we can
> easily remove while simplifying the code at the same time.
>
> For 2), I agree that at a certain scale or for some scenarios, some
> features simply do not scale or catch users by surprise. The goal of the
> CEP is to improve things in 2 ways. One is by making Cassandra smarter in
> the way it chooses how to process queries, hopefully improving its overall
> scalability. The other by being transparent about how Cassandra will
> execute the queries through the use of EXPLAIN. One problem of GROUP BY for
> example is that most users do not realize what is actually happening under
> the hood and therefore its limitations. I do not believe that EXPLAIN will
> change everything but it will help people to get a better understanding of
> the limitations of some features.
>
> I do not know which features will be added in the future to C*. That will
> be discussed through some future CEPs. Nevertheless, I do not believe that
> it makes sense to write a CEP for a query optimizer without taking into
> account that we might at some point add some level of support for joins or
> subqueries. We have been too often delivering features without looking at
> what could be the possible evolutions which resulted in code where adding
> new features was more complex than it should have been. I do not want to
> make the same mistake. I want to create an optimizer that can be improved
> easily and considering joins or other features simply help to build things
> in a more generic way.
>
> Regarding feature stabilization, I believe that it is happening. I have
> heard plans of how to solve MVs, range queries, hot partitions, ... and
> there was a lot of thinking behind those plans. Secondary indexes are being
> worked on. We hope that the optimizer will also help with some index
> queries.
>
> It seems to me that this proposal is going toward the direction that you
> want without introducing new problems for scalability.
>
>
>
>
> Le jeu. 14 déc. 2023 à 16:47, Chris Lohfink  a
> écrit :
>
> I don't wanna be a blocker for this CEP or anything but did want to put my
> 2 cents in. This CEP is horrifying to me.
>
> I have seen thousands of clusters across multiple companies and helped
> them get working successfully. A vast majority of that involved blocking
> the use of MVs, GROUP BY, secondary indexes, and even just simple _range
> queries_. The "unncessary restrictions of cql" are not only necessary IMHO,
> more restrictions are necessary to be successful at scale. The idea of just
> opening up CQL to general purpose relational queries and lines like 
> "supporting
> queries with joins in an efficient way" ... I would really like us to
> make secondary indexes be a viable option before we start opening up
> floodgates on stuff like this.
>
> Chris
>
> On Thu, Dec 14, 2023 at 9:37 AM Benedict  wrote:
>
>
> > So yes, this physical plan is the structure that you have in mind but
> the idea of

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

>
> If we are to address that within the CEP itself then we should discuss it
> here, as I would like to fully understand the approach as well as how it
> relates to consistency of execution and the idea of triggering
> re-optimisation.


Sure, that was my plan.

I’m not sold on the proposed set of characteristics, and think my coupling
> an execution plan to a given prepared statement for clients to supply is
> perhaps simpler to implement and maintain, and has corollary benefits -
> such as providing a mechanism for users to specify their own execution plan.
>

>
Note, my proposal cuts across all of these elements of the CEP. There is no
> obvious need for a cross-cluster re-optimisation event or cross cluster
> statistic management.
>

I think that I am missing one part of your proposal. How do you plan to
build the initial execution plan for a prepared statement?

Le mer. 20 déc. 2023 à 14:05, Benedict  a écrit :

> If we are to address that within the CEP itself then we should discuss it
> here, as I would like to fully understand the approach as well as how it
> relates to consistency of execution and the idea of triggering
> re-optimisation. These ideas are all interrelated.
>
> I’m not sold on the proposed set of characteristics, and think my coupling
> an execution plan to a given prepared statement for clients to supply is
> perhaps simpler to implement and maintain, and has corollary benefits -
> such as providing a mechanism for users to specify their own execution plan.
>
> Note, my proposal cuts across all of these elements of the CEP. There is
> no obvious need for a cross-cluster re-optimisation event or cross cluster
> statistic management.
>
> We still also need to discuss more concretely how the base statistics
> themselves will be derived, as there is little detail here today in the
> proposal.
>
> On 20 Dec 2023, at 12:58, Benjamin Lerer  wrote:
>
> 
> After the second phase of the CEP, we will have two optimizer
> implementations. One will be similar to what we have today and the other
> one will be the CBO. As those implementations will be behind the new
> Optimizer API interfaces they will both have support for EXPLAIN and they
> will both benefit from the simplification/normalization rules. Such as the
> ones that David mentioned.
>
> Regarding functions, we are already able to determine which ones are
> deterministic (
> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/Function.java#L55).
> We simply do not take advantage of it.
>
> I removed the ALLOW FILTERING part and will open a discussion about it at
> the beginning of next year.
>
> Regarding the statistics management part, I would like to try to address
> it within the CEP itself, if feasible. If it turns out to be too
> complicated, I will separate it into its own CEP.
>
> Le mar. 19 déc. 2023 à 22:23, David Capwell  a écrit :
>
>> even if the only outcome of all this work were to tighten up
>> inconsistencies in our grammar and provide more robust EXPLAIN and EXPLAIN
>> ANALYZE functionality to our end users, I think that would be highly
>> valuable
>>
>>
>> In my mental model a no-op optimizer just becomes what we have today
>> (since all new features really should be disabled by default, I would hope
>> we support this), so we benefit from having a logical AST + ability to
>> mutate it before we execute it and we can use this to make things nicer for
>> users (as you are calling out)
>>
>> Here is one example that stands out to me in accord
>>
>> LET a = (select * from tbl where pk=0);
>> Insert into tbl2 (pk, …) values (a.pk, …); — this is not allowed as we
>> don’t know the primary key… but this could trivially be written to replace
>> a.pk with 0…
>>
>> With this work we could also rethink what functions are deterministic and
>> which ones are not (not trying to bike shed)… simple example is “now”
>> (select now() from tbl; — each row will have a different timestamp), if we
>> make this deterministic we can avoid calling it for each row and instead
>> just replace it with a constant for the query…
>>
>> Even if the CBO is dropped in favor of no-op (what we do today), I still
>> see value in this work.
>>
>> I do think that the CBO really doesn’t solve the fact some features don’t
>> work well, if anything it could just mask it until it’s too late….  If user
>> builds an app using filtering and everything is going well in QA, but once
>> they see a spike in traffic in prod we start rejecting… this is a bad
>> user experience IMO… we KNOW you must think about this before you go this
>> route, so a CBO letting yo

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

>
> Pick a random replica and ask it to prepare it, and use that. This is
> probably fine, unless there is significant skew (in which case, our plan is
> likely to bad whatever, somewhere)


To be sure that I understand you correctly. What you are suggesting is to
use local statistics to compute the execution plan and then rely on the
driver performing the query to cache it?

If it is the case, then it would mean that:

   - a bad execution plan will only be corrected on client restart
   - if the data distribution change the driver will keep on using its plan
   - different drivers might use different plans
   - 2 similar queries that differ on simple ways might have totally
   different plans

Do I understand your proposal and its trades off correctly?

Le mer. 20 déc. 2023 à 17:16, Benedict  a écrit :

> I see three options:
>
>1. Pick a random replica and ask it to prepare it, and use that. This
>is probably fine, unless there is significant skew (in which case, our plan
>is likely to bad whatever, somewhere)
>2. If there already exists a plan for the query, return that
>3. Pick a sample of replicas and either
>   1. Ask them for their plan, and pick the most common plan
>   2. Ask them for their cardinality estimates, and use some
>   combination thereof
>
> The nice thing is that this can be evolved independently of the rest of
> the proposal, as the simple option is fine. But also we don’t have to
> introduce any new whole cluster operations - any work we might perform is
> done only on statement preparation, and only for data relevant to the
> statement that is being prepared, and only on a minority of shardsand one
> replica per shard (in a single DC), rather than on some arbitrary schedule
> for every replica.
>
> On 20 Dec 2023, at 15:52, Benjamin Lerer  wrote:
>
> 
>
>> If we are to address that within the CEP itself then we should discuss it
>> here, as I would like to fully understand the approach as well as how it
>> relates to consistency of execution and the idea of triggering
>> re-optimisation.
>
>
> Sure, that was my plan.
>
> I’m not sold on the proposed set of characteristics, and think my coupling
>> an execution plan to a given prepared statement for clients to supply is
>> perhaps simpler to implement and maintain, and has corollary benefits -
>> such as providing a mechanism for users to specify their own execution plan.
>>
>
>>
> Note, my proposal cuts across all of these elements of the CEP. There is
>> no obvious need for a cross-cluster re-optimisation event or cross cluster
>> statistic management.
>>
>
> I think that I am missing one part of your proposal. How do you plan to
> build the initial execution plan for a prepared statement?
>
> Le mer. 20 déc. 2023 à 14:05, Benedict  a écrit :
>
>> If we are to address that within the CEP itself then we should discuss it
>> here, as I would like to fully understand the approach as well as how it
>> relates to consistency of execution and the idea of triggering
>> re-optimisation. These ideas are all interrelated.
>>
>> I’m not sold on the proposed set of characteristics, and think my
>> coupling an execution plan to a given prepared statement for clients to
>> supply is perhaps simpler to implement and maintain, and has corollary
>> benefits - such as providing a mechanism for users to specify their own
>> execution plan.
>>
>> Note, my proposal cuts across all of these elements of the CEP. There is
>> no obvious need for a cross-cluster re-optimisation event or cross cluster
>> statistic management.
>>
>> We still also need to discuss more concretely how the base statistics
>> themselves will be derived, as there is little detail here today in the
>> proposal.
>>
>> On 20 Dec 2023, at 12:58, Benjamin Lerer  wrote:
>>
>> 
>> After the second phase of the CEP, we will have two optimizer
>> implementations. One will be similar to what we have today and the other
>> one will be the CBO. As those implementations will be behind the new
>> Optimizer API interfaces they will both have support for EXPLAIN and they
>> will both benefit from the simplification/normalization rules. Such as the
>> ones that David mentioned.
>>
>> Regarding functions, we are already able to determine which ones are
>> deterministic (
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/Function.java#L55).
>> We simply do not take advantage of it.
>>
>> I removed the ALLOW FILTERING part and will open a discussion about it at
>> the beginning of next year.
>>
>> Regarding 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

est one I can think of right now -
> e.g., if we wanted to add the ability to issue partition-restricted queries
> over a base table with multiple indexes defined without users specifically
> declaring an index. I haven’t seen an at-scale use case that would be
> better served by planner-driven index selection vs. user-driven, but they
> might be out there.
>
> It’s not my role to suggest changes in prioritization for work that isn’t
> mine. But I feel that the project could design better interfaces and a
> better planner/optimizer if that work were oriented toward improving
> particular features that are in wide use.
>
> To summarize my thoughts:
>
> – I agree that it is valuable for Apache Cassandra to gain a
> planner/optimizer.
> – I disagree with removing or deprecating ALLOW FILTERING and see this as
> a necessary guardrail.
> – I think the proposal surfaces the differences between the design goals
> of CQL and SQL, but I don’t feel that it quite addresses it.
> – I think we could collectively build a stronger planner/optimizer once
> some of the features it’s meant to optimize are in place.
> – I’m not quite sold on the need for the implementation to be bespoke
> based on discussion so far (vs. Calcite/Catalyst etc), but haven’t done the
> legwork to investigate this myself.
> – I *love* the idea of capturing many of the execution and hotness
> statistics that are proposed in the CEP. It would be very valuable to
> surface query cost to users independent of a CBO. Stats like these would
> also be valuable toward retrofitting Cassandra for multitenancy by bounding
> or rate-limiting users on query cost. Tracking SSTable hotness would also
> be useful toward evaluating feasibility of tiered storage, too.
>
> Thanks for this proposal and discussion so far — appreciate and enjoying
> it.
>
> – Scott
>
> On Dec 20, 2023, at 7:52 AM, Benjamin Lerer  wrote:
>
>
> If we are to address that within the CEP itself then we should discuss it
>> here, as I would like to fully understand the approach as well as how it
>> relates to consistency of execution and the idea of triggering
>> re-optimisation.
>>
>
> Sure, that was my plan.
>
>
> I’m not sold on the proposed set of characteristics, and think my coupling
>> an execution plan to a given prepared statement for clients to supply is
>> perhaps simpler to implement and maintain, and has corollary benefits -
>> such as providing a mechanism for users to specify their own execution plan.
>>
>
>>
> Note, my proposal cuts across all of these elements of the CEP. There is
>> no obvious need for a cross-cluster re-optimisation event or cross cluster
>> statistic management.
>>
>
> I think that I am missing one part of your proposal. How do you plan to
> build the initial execution plan for a prepared statement?
>
> Le mer. 20 déc. 2023 à 14:05, Benedict  a écrit :
>
>>
>> If we are to address that within the CEP itself then we should discuss it
>> here, as I would like to fully understand the approach as well as how it
>> relates to consistency of execution and the idea of triggering
>> re-optimisation. These ideas are all interrelated.
>>
>> I’m not sold on the proposed set of characteristics, and think my
>> coupling an execution plan to a given prepared statement for clients to
>> supply is perhaps simpler to implement and maintain, and has corollary
>> benefits - such as providing a mechanism for users to specify their own
>> execution plan.
>>
>> Note, my proposal cuts across all of these elements of the CEP. There is
>> no obvious need for a cross-cluster re-optimisation event or cross cluster
>> statistic management.
>>
>> We still also need to discuss more concretely how the base statistics
>> themselves will be derived, as there is little detail here today in the
>> proposal.
>>
>>
>> On 20 Dec 2023, at 12:58, Benjamin Lerer  wrote:
>>
>> 
>> After the second phase of the CEP, we will have two optimizer
>> implementations. One will be similar to what we have today and the other
>> one will be the CBO. As those implementations will be behind the new
>> Optimizer API interfaces they will both have support for EXPLAIN and they
>> will both benefit from the simplification/normalization rules. Such as the
>> ones that David mentioned.
>>
>> Regarding functions, we are already able to determine which ones are
>> deterministic (
>> https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/cql3/functions/Function.java#L55).
>> We simply do not take advantage of it.
>>
>> I removed the ALLOW FILTE

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benjamin Lerer]

n the coming years, Cassandra is likely to gain
> semi-relational features via maturation of the byte-ordered partitioner
> (via range splitting, via TCM); the availability of SAI and its evolution
> (e.g., via new functionality enabled by Lucene libraries); potentially
> joins via BOP; and others. This is a really exciting future, and one that
> probably requires a planner and optimizer.
>
> My general inclination is that a planner + optimizer seem valuable for
> Cassandra, but that the proposal feels a year or two early. The database
> doesn’t yet have a plan of record to add support for some of the
> semirelational constructs we’ve talked about, and I’m not aware of active
> CEPs that propose designs for features like these yet.
>
> Like Jeff, I’d find this much easier to discuss in the context of a
> database gaining support for these features with specific designs available
> to discuss. The ALLOW FILTERING and constant folding examples are a little
> slim. Index selection is probably the best one I can think of right now -
> e.g., if we wanted to add the ability to issue partition-restricted queries
> over a base table with multiple indexes defined without users specifically
> declaring an index. I haven’t seen an at-scale use case that would be
> better served by planner-driven index selection vs. user-driven, but they
> might be out there.
>
> It’s not my role to suggest changes in prioritization for work that isn’t
> mine. But I feel that the project could design better interfaces and a
> better planner/optimizer if that work were oriented toward improving
> particular features that are in wide use.
>
> To summarize my thoughts:
>
> – I agree that it is valuable for Apache Cassandra to gain a
> planner/optimizer.
> – I disagree with removing or deprecating ALLOW FILTERING and see this as
> a necessary guardrail.
> – I think the proposal surfaces the differences between the design goals
> of CQL and SQL, but I don’t feel that it quite addresses it.
> – I think we could collectively build a stronger planner/optimizer once
> some of the features it’s meant to optimize are in place.
> – I’m not quite sold on the need for the implementation to be bespoke
> based on discussion so far (vs. Calcite/Catalyst etc), but haven’t done the
> legwork to investigate this myself.
> – I *love* the idea of capturing many of the execution and hotness
> statistics that are proposed in the CEP. It would be very valuable to
> surface query cost to users independent of a CBO. Stats like these would
> also be valuable toward retrofitting Cassandra for multitenancy by bounding
> or rate-limiting users on query cost. Tracking SSTable hotness would also
> be useful toward evaluating feasibility of tiered storage, too.
>
> Thanks for this proposal and discussion so far — appreciate and enjoying
> it.
>
> – Scott
>
> On Dec 20, 2023, at 7:52 AM, Benjamin Lerer  wrote:
>
>
> If we are to address that within the CEP itself then we should discuss it
> here, as I would like to fully understand the approach as well as how it
> relates to consistency of execution and the idea of triggering
> re-optimisation.
>
>
> Sure, that was my plan.
>
>
> I’m not sold on the proposed set of characteristics, and think my coupling
> an execution plan to a given prepared statement for clients to supply is
> perhaps simpler to implement and maintain, and has corollary benefits -
> such as providing a mechanism for users to specify their own execution plan.
>
>
>
> Note, my proposal cuts across all of these elements of the CEP. There is
> no obvious need for a cross-cluster re-optimisation event or cross cluster
> statistic management.
>
>
> I think that I am missing one part of your proposal. How do you plan to
> build the initial execution plan for a prepared statement?
>
> Le mer. 20 déc. 2023 à 14:05, Benedict  a écrit :
>
>
> If we are to address that within the CEP itself then we should discuss it
> here, as I would like to fully understand the approach as well as how it
> relates to consistency of execution and the idea of triggering
> re-optimisation. These ideas are all interrelated.
>
> I’m not sold on the proposed set of characteristics, and think my coupling
> an execution plan to a given prepared statement for clients to supply is
> perhaps simpler to implement and maintain, and has corollary benefits -
> such as providing a mechanism for users to specify their own execution plan.
>
> Note, my proposal cuts across all of these elements of the CEP. There is
> no obvious need for a cross-cluster re-optimisation event or cross cluster
> statistic management.
>
> We still also need to discuss more concretely how the base statistics
> themselves will be

Re: Welcome Maxim Muzafarov as Cassandra Committer

[Benjamin Lerer]

I am always late to the party. ;-)
 Congrats Maxim!

Le mar. 9 janv. 2024 à 13:16, Maxim Muzafarov  a écrit :

> Thank you all so much, I'm happy to be part of such an active
> community and to be able to contribute to the product that is used all
> over the world!
>
> On Tue, 9 Jan 2024 at 12:33, Mike Adamson  wrote:
> >
> > Congrats Maxim!!
> >
> > On Tue, 9 Jan 2024, 10:41 Andrés de la Peña, 
> wrote:
> >>
> >> Congrats, Maxim!
> >>
> >> On Tue, 9 Jan 2024 at 03:45, guo Maxwell  wrote:
> >>>
> >>> Congratulations, Maxim!
> >>>
> >>> Francisco Guerrero  于2024年1月9日周二 09:00写道：
> 
>  Congratulations, Maxim! Well deserved!
> 
>  On 2024/01/08 18:19:04 Josh McKenzie wrote:
>  > The Apache Cassandra PMC is pleased to announce that Maxim
> Muzafarov has accepted
>  > the invitation to become a committer.
>  >
>  > Thanks for all the hard work and collaboration on the project thus
> far, and we're all looking forward to working more with you in the future.
> Congratulations and welcome!
>  >
>  > The Apache Cassandra PMC members
>  >
>  >
>

[DISCUSS] NULL handling and the unfrozen collection issue

[Benjamin Lerer]

Hi everybody,

CEP-29 (CQL NOT Operator) is hitting the grey area of how we want as a
community to handle NULL including for things like unfrozen (multi-cell)
collections and I would like to make a proposal for moving forward with
NULL related issues.

We have currently 2 tickets open about NULL handling (I might have missed
others):

   1. CASSANDRA-10715
   : Allowing
   Filtering on NULL
   2. CASSANDRA-17762
   : LWT IF col =
   NULL is inconsistent with SQL NULL

We also had previously some discussion on which we touched the subject:

   - [DISCUSS] LWT UPDATE semantics with + and - when null
   - CEP-15 multi key transaction syntax

In all those tickets and discussions the consensus was to have a behavior
similar to SQL.

For null comparisons, SQL uses the three-value logic (
https://modern-sql.com/concept/three-valued-logic) introducing the need for
IS NULL and IS NOT NULL operators. Those conflict with the col = NULL
predicate supported in LWT conditions (CASSANDRA-17762
).

So far, as Cassandra was only using inclusive operators, comparisons were
behaving in an expected way. According to three-valued logic NULL CONTAINS
'foo' should return UNKNOWN and the filtering behavior should exclude
everything which is not true.Therefore the row should not be returned as
expected. With exclusive operators things are more tricky. NULL NOT
CONTAINS 'foo' will also return UNKNOWN causing the row to not be returned
which might not match people's expectations.
This behavior can be even more confusing once you take into account empty
and null collections. NOT CONTAINS on an empty collection will return true
while it will return UNKNOWN on a NULL collection. Unfortunately, for
unfrozen (multicell) collections we are unable to differentiate between an
empty and null collection and therefore always treat empty collections as
NULL.
For predicates such as map[myKey] != 'foo' when myKey is not present the
result can also be surprising as it will end up comparing NULL to 'foo'
returning once more UNKNOWN and ignoring the row.
In order to respect the SQL three-valued logic and be able to allow the
user to fetch all the rows which do not contains a specific value we would
need support IS NULL, IS NOT NULL and OR to allow query like:
WHERE c IS NULL OR c NOT CONTAINS 'foo' / WHERE m IS NULL OR m[myKey] != foo

Supporting the three-valued logic makes sense to me even if some behavior
might end up being confusing. In which case we can easily fix
CASSANDRA-10715 and deprectate support for col = NULL/col != NULL in LWT.

What is people's opinion? Should we go for the three-valued logic
everywhere? Should we try something else?

[DISCUSS] CQL handling of WHERE clause predicates

[Benjamin Lerer]

Hi everybody,

CQL appears to be inconsistent in how it handles predicates.

One type of inconsistencies is that some operators can be used in some
places but not in others or on some expressions but not others.
For example:

   - != can be used in LWT conditions but not elsewhere
   - Token expressions (e.g. token(pk) = ?) support =, >, >=, =< and < but
   not IN
   - Map elements predicates (e.g m[?] = ?) only support =
   - ... (a long list)

This type of inconsistencies can be easily fixed over time.

The other type of inconsistencies that is more confusing is about how we
deal with the combination of multiple predicates as we accept some and
reject others.
For example, we allow: "token(pk) > ? AND pk IN ?" but reject "c > ? AND  c
IN ?".
For CQL, rejection seems to be the norm with only a few exceptions. Whereas
SQL accepts all inputs in terms of predicates combination.

For the IS NULL and IS NOT NULL predicates for partition Key and clustering
columns, that we know cannot be null, that lead us to 2 choices: either
throwing an exception when any of them is specified on partition
keys/clustering columns or accepting them. Ignoring IS NOT NULL as it is a
noop and returning no rows when IS NULL is specified.

I personally prefer the SQL behavior which can result in a simpler code.
Even if I can also understand the defensive approach.

One way or the other, I believe that it would be good to standardize on one
approach and would like to hear your opinion.

Re: [DISCUSS] NULL handling and the unfrozen collection issue

[Benjamin Lerer]

>
> Now, in Cassandra setting a column to null means deleting it and if *all*
> columns in a row are null the row is deleted. This might be another edge
> case...


It is slightly more complicated than that as the primary key columns count
in (*all* columns)

For example if you have the following table: CREATE TABLE tlb (pk int, c
int, v int, PRIMARY KEY (pk, c)) and the following row: INSERT INTO tlb
(pk, c, v) VALUES (1, 1, 1)
deleting the column v (DELETE v FROM %s WHERE pk = 1 AND c = 1) will not
delete the row as the primary key columns have a timestamp and therefore do
exist. So the row will still exist with a null value for column v.

If the row was created through an UPDATE (UPDATE tlb SET v = 1 WHERE pk = 1
AND c = 1) things will be different as an UPDATE statement do not create a
timestamp for the clustering columns. By consequence, if V is deleted (set
to null) the row will not have any columns anymore and will be deleted.

The issue here is that in practice we never consider partition keys or
clustering columns as null if the database returns a row for it. Whether
the primary key columns have a timestamp or not.
I believe that we should ignore that fact too as far as IS NULL/IS NOT NULL
are concerned. If a row exists, its primary columns should be considered as
not null. Otherwise we are getting on a really slippery slope. The
INSERT/UPDATE logic is confusing enough in my opinion without adding
another layer to it.

One other issue that we have though is that the code for != LWT does not
work with the Three-Valued logic. If you have: [...] IF col != 1  and col
is null then in the TVL the value should be UNKNOWN therefore the condition
should not match.
It feels to me that we should probably keep that behavior for backward
compatibility reasons but probably change the behavior in Accord if it is
not already done.



Le jeu. 21 mars 2024 à 01:10, German Eichberger via dev <
dev@cassandra.apache.org> a écrit :

> Hi,
>
> +1 I like doing it the SQL way. This makes sense to me.
>
> Now, in Cassandra setting a column to null means deleting it and if *all*​
> columns in a row are null the row is deleted. This might be another edge
> case...
>
> German
> ------
> *From:* Benjamin Lerer 
> *Sent:* Wednesday, March 20, 2024 9:15 AM
> *To:* dev@cassandra.apache.org 
> *Subject:* [EXTERNAL] [DISCUSS] NULL handling and the unfrozen collection
> issue
>
> You don't often get email from b.le...@gmail.com. Learn why this is
> important <https://aka.ms/LearnAboutSenderIdentification>
> Hi everybody,
>
> CEP-29 (CQL NOT Operator) is hitting the grey area of how we want as a
> community to handle NULL including for things like unfrozen (multi-cell)
> collections and I would like to make a proposal for moving forward with
> NULL related issues.
>
> We have currently 2 tickets open about NULL handling (I might have missed
> others):
>
>1. CASSANDRA-10715
><https://issues.apache.org/jira/browse/CASSANDRA-10715>: Allowing
>Filtering on NULL
>2. CASSANDRA-17762
><https://issues.apache.org/jira/browse/CASSANDRA-17762>: LWT IF col =
>NULL is inconsistent with SQL NULL
>
> We also had previously some discussion on which we touched the subject:
>
>- [DISCUSS] LWT UPDATE semantics with + and - when null
>- CEP-15 multi key transaction syntax
>
> In all those tickets and discussions the consensus was to have a behavior
> similar to SQL.
>
> For null comparisons, SQL uses the three-value logic (
> https://modern-sql.com/concept/three-valued-logic) introducing the need
> for IS NULL and IS NOT NULL operators. Those conflict with the col = NULL
> predicate supported in LWT conditions (CASSANDRA-17762
> <https://issues.apache.org/jira/browse/CASSANDRA-17762>).
>
> So far, as Cassandra was only using inclusive operators, comparisons were
> behaving in an expected way. According to three-valued logic NULL CONTAINS
> 'foo' should return UNKNOWN and the filtering behavior should exclude
> everything which is not true.Therefore the row should not be returned as
> expected. With exclusive operators things are more tricky. NULL NOT
> CONTAINS 'foo' will also return UNKNOWN causing the row to not be returned
> which might not match people's expectations.
> This behavior can be even more confusing once you take into account empty
> and null collections. NOT CONTAINS on an empty collection will return true
> while it will return UNKNOWN on a NULL collection. Unfortunately, for
> unfrozen (multicell) collections we are unable to differentiate between an
> empty and null collection and therefore always treat empty collections as
> NULL.
> For predicates such as map[myKey] != 'foo' when myKey is not present the
> 

Welcome Alexandre Dutra, Andrew Tolbert, Bret McGuire, Olivier Michallat as Cassandra Committers

[Benjamin Lerer]

 The Apache Cassandra PMC is pleased to announce that Alexandre Dutra,
Andrew Tolbert, Bret McGuire and Olivier Michallat have accepted the
invitation to become committers on the java driver sub-project.

Thanks for your contributions to the Java driver during all those years!
Congratulations and welcome!

The Apache Cassandra PMC members

[DISCUSS] Adding support for BETWEEN operator

[Benjamin Lerer]

Hi everybody,

Just raising awareness that Simon is working on adding support for the
BETWEEN operator in WHERE clauses (SELECT and DELETE) in CASSANDRA-19604.
We plan to add support for it in conditions in a separate patch.

The patch is available.

As a side note, Simon chose to do his highschool senior project
contributing to Apache Cassandra. This patch is his first contribution for
his senior project (his second feature contribution to Apache Cassandra).

Re: [DISCUSS] Adding support for BETWEEN operator

[Benjamin Lerer]

Range restrictions (>, >=, =<, < and BETWEEN) do not work on UPDATEs. They
do work on DELETE because under the hood C* they get translated into range
tombstones.

Le mar. 14 mai 2024 à 02:44, David Capwell  a écrit :

> I would also include in UPDATE… but yeah, <3 BETWEEN and welcome this work.
>
> On May 13, 2024, at 7:40 AM, Patrick McFadin  wrote:
>
> This is a great feature addition to CQL! I get asked about it from time to
> time but then people figure out a workaround. It will be great to just have
> it available.
>
> And right on Simon! I think the only project I had as a high school senior
> was figuring out how many parties I could go to and still maintain a
> passing grade. Thanks for your work here.
>
> Patrick
>
> On Mon, May 13, 2024 at 1:35 AM Benjamin Lerer  wrote:
>
>> Hi everybody,
>>
>> Just raising awareness that Simon is working on adding support for the
>> BETWEEN operator in WHERE clauses (SELECT and DELETE) in CASSANDRA-19604.
>> We plan to add support for it in conditions in a separate patch.
>>
>> The patch is available.
>>
>> As a side note, Simon chose to do his highschool senior project
>> contributing to Apache Cassandra. This patch is his first contribution for
>> his senior project (his second feature contribution to Apache Cassandra).
>>
>>
>>
>

Re: [DISCUSS] Adding support for BETWEEN operator

[Benjamin Lerer]

It should be like range tombstones ... in much worse ;-). A tombstone is a
simple marker (deleted). An update can be far more complex.

Le mar. 14 mai 2024 à 15:52, Jon Haddad  a écrit :

> Is there a technical limitation that would prevent a range write that
> functions the same way as a range tombstone, other than probably needing a
> version bump of the storage format?
>
>
> On Tue, May 14, 2024 at 12:03 AM Benjamin Lerer  wrote:
>
>> Range restrictions (>, >=, =<, < and BETWEEN) do not work on UPDATEs.
>> They do work on DELETE because under the hood C* they get translated into
>> range tombstones.
>>
>> Le mar. 14 mai 2024 à 02:44, David Capwell  a écrit :
>>
>>> I would also include in UPDATE… but yeah, <3 BETWEEN and welcome this
>>> work.
>>>
>>> On May 13, 2024, at 7:40 AM, Patrick McFadin  wrote:
>>>
>>> This is a great feature addition to CQL! I get asked about it from time
>>> to time but then people figure out a workaround. It will be great to just
>>> have it available.
>>>
>>> And right on Simon! I think the only project I had as a high school
>>> senior was figuring out how many parties I could go to and still maintain a
>>> passing grade. Thanks for your work here.
>>>
>>> Patrick
>>>
>>> On Mon, May 13, 2024 at 1:35 AM Benjamin Lerer 
>>> wrote:
>>>
>>>> Hi everybody,
>>>>
>>>> Just raising awareness that Simon is working on adding support for the
>>>> BETWEEN operator in WHERE clauses (SELECT and DELETE) in CASSANDRA-19604.
>>>> We plan to add support for it in conditions in a separate patch.
>>>>
>>>> The patch is available.
>>>>
>>>> As a side note, Simon chose to do his highschool senior project
>>>> contributing to Apache Cassandra. This patch is his first contribution for
>>>> his senior project (his second feature contribution to Apache Cassandra).
>>>>
>>>>
>>>>
>>>

Re: [DISCUSS] Adding support for BETWEEN operator

[Benjamin Lerer]

>
> It would be a lot more constructive to apply our brains towards solving an
> interesting problem than pointing out all its potential flaws based on gut
> feelings.


It is not simply a gut feeling, Jon. This change impacts read, write,
indexing, storage, compaction, repair... The risk and cost associated with
it are pretty significant and I am not convinced at this point of its
benefit.

Le mar. 14 mai 2024 à 19:05, Jon Haddad  a écrit :

> Personally, I don't think that something being scary at first glance is a
> good reason not to explore an idea.  The scenario you've described here is
> tricky but I'm not expecting it to be any worse than say, SAI, which (the
> last I checked) has O(N) complexity on returning result sets with regard to
> rows returned.  We've also merged in Vector search which has O(N) overhead
> with the number of SSTables.  We're still fundamentally looking at, in most
> cases, a limited number of SSTables and some merging of values.
>
> Write updates are essentially a timestamped mask, potentially overlapping,
> and I suspect potentially resolvable during compaction by propagating the
> values.  They could be eliminated or narrowed based on how they've
> propagated by using the timestamp metadata on the SSTable.
>
> It would be a lot more constructive to apply our brains towards solving an
> interesting problem than pointing out all its potential flaws based on gut
> feelings.  We haven't even moved this past an idea.
>
> I think it would solve a massive problem for a lot of people and is 100%
> worth considering.  Thanks Patrick and David for raising this.
>
> Jon
>
>
>
> On Tue, May 14, 2024 at 9:48 AM Bowen Song via dev <
> dev@cassandra.apache.org> wrote:
>
>> Ranged update sounds like a disaster for compaction and read performance.
>>
>> Imagine compacting or reading some SSTables in which a large number of
>> overlapping but non-identical ranges were updated with different values. It
>> gives me a headache by just thinking about it.
>>
>> Ranged delete is much simpler, because the "value" is the same tombstone
>> marker, and it also is guaranteed to expire and disappear eventually, so
>> the performance impact of dealing with them at read and compaction time
>> doesn't suffer in the long term.
>>
>> On 14/05/2024 16:59, Benjamin Lerer wrote:
>>
>> It should be like range tombstones ... in much worse ;-). A tombstone is
>> a simple marker (deleted). An update can be far more complex.
>>
>> Le mar. 14 mai 2024 à 15:52, Jon Haddad  a écrit :
>>
>>> Is there a technical limitation that would prevent a range write that
>>> functions the same way as a range tombstone, other than probably needing a
>>> version bump of the storage format?
>>>
>>>
>>> On Tue, May 14, 2024 at 12:03 AM Benjamin Lerer 
>>> wrote:
>>>
>>>> Range restrictions (>, >=, =<, < and BETWEEN) do not work on UPDATEs.
>>>> They do work on DELETE because under the hood C* they get translated into
>>>> range tombstones.
>>>>
>>>> Le mar. 14 mai 2024 à 02:44, David Capwell  a
>>>> écrit :
>>>>
>>>>> I would also include in UPDATE… but yeah, <3 BETWEEN and welcome this
>>>>> work.
>>>>>
>>>>> On May 13, 2024, at 7:40 AM, Patrick McFadin 
>>>>> wrote:
>>>>>
>>>>> This is a great feature addition to CQL! I get asked about it from
>>>>> time to time but then people figure out a workaround. It will be great to
>>>>> just have it available.
>>>>>
>>>>> And right on Simon! I think the only project I had as a high school
>>>>> senior was figuring out how many parties I could go to and still maintain 
>>>>> a
>>>>> passing grade. Thanks for your work here.
>>>>>
>>>>> Patrick
>>>>>
>>>>> On Mon, May 13, 2024 at 1:35 AM Benjamin Lerer 
>>>>> wrote:
>>>>>
>>>>>> Hi everybody,
>>>>>>
>>>>>> Just raising awareness that Simon is working on adding support for
>>>>>> the BETWEEN operator in WHERE clauses (SELECT and DELETE) in
>>>>>> CASSANDRA-19604. We plan to add support for it in conditions in a 
>>>>>> separate
>>>>>> patch.
>>>>>>
>>>>>> The patch is available.
>>>>>>
>>>>>> As a side note, Simon chose to do his highschool senior project
>>>>>> contributing to Apache Cassandra. This patch is his first contribution 
>>>>>> for
>>>>>> his senior project (his second feature contribution to Apache Cassandra).
>>>>>>
>>>>>>
>>>>>>
>>>>>

Re: [DISCUSS] NULL handling and the unfrozen collection issue

[Benjamin Lerer]

I found some other confusing behavior in LWT around null value and empty
multicell collection. I opened CASSANDRA-19637
<https://issues.apache.org/jira/browse/CASSANDRA-19637>for those interested.

Le jeu. 4 avr. 2024 à 18:34, Caleb Rackliffe  a
écrit :

> The easiest way to check out how Accord uses IS NULL and IS NOT NULL is to
> look at the examples in the cep-15-accord branch:
>
>
> https://github.com/apache/cassandra/blob/cep-15-accord/test/distributed/org/apache/cassandra/distributed/test/accord/AccordCQLTestBase.java
>
> tl;dr We did indeed try to go with an approach that more closely matches
> SQL, although there may still be some edges we didn't test.
>
> I'd have no problem w/ moving to 3-value logic everywhere, I guess, but
> "everywhere" could just mean filtering queries and Accord. (i.e. If we want
> to deprecate LWT col = NULL syntax, do we really want people rewriting
> those LWTs...or just moving to the new Accord syntax, which obviously
> supports it? We should "spend" our user query rewrite budget wisely.)
>
> On Thu, Apr 4, 2024 at 4:53 AM Benjamin Lerer  wrote:
>
>> Now, in Cassandra setting a column to null means deleting it and if *all*
>>> columns in a row are null the row is deleted. This might be another edge
>>> case...
>>
>>
>> It is slightly more complicated than that as the primary key columns
>> count in (*all* columns)
>>
>> For example if you have the following table: CREATE TABLE tlb (pk int, c
>> int, v int, PRIMARY KEY (pk, c)) and the following row: INSERT INTO tlb
>> (pk, c, v) VALUES (1, 1, 1)
>> deleting the column v (DELETE v FROM %s WHERE pk = 1 AND c = 1) will not
>> delete the row as the primary key columns have a timestamp and therefore do
>> exist. So the row will still exist with a null value for column v.
>>
>> If the row was created through an UPDATE (UPDATE tlb SET v = 1 WHERE pk =
>> 1 AND c = 1) things will be different as an UPDATE statement do not create
>> a timestamp for the clustering columns. By consequence, if V is deleted
>> (set to null) the row will not have any columns anymore and will be deleted.
>>
>> The issue here is that in practice we never consider partition keys or
>> clustering columns as null if the database returns a row for it. Whether
>> the primary key columns have a timestamp or not.
>> I believe that we should ignore that fact too as far as IS NULL/IS NOT
>> NULL are concerned. If a row exists, its primary columns should be
>> considered as not null. Otherwise we are getting on a really slippery
>> slope. The INSERT/UPDATE logic is confusing enough in my opinion without
>> adding another layer to it.
>>
>> One other issue that we have though is that the code for != LWT does not
>> work with the Three-Valued logic. If you have: [...] IF col != 1  and col
>> is null then in the TVL the value should be UNKNOWN therefore the condition
>> should not match.
>> It feels to me that we should probably keep that behavior for backward
>> compatibility reasons but probably change the behavior in Accord if it is
>> not already done.
>>
>>
>>
>> Le jeu. 21 mars 2024 à 01:10, German Eichberger via dev <
>> dev@cassandra.apache.org> a écrit :
>>
>>> Hi,
>>>
>>> +1 I like doing it the SQL way. This makes sense to me.
>>>
>>> Now, in Cassandra setting a column to null means deleting it and if
>>> *all*​ columns in a row are null the row is deleted. This might be
>>> another edge case...
>>>
>>> German
>>> --
>>> *From:* Benjamin Lerer 
>>> *Sent:* Wednesday, March 20, 2024 9:15 AM
>>> *To:* dev@cassandra.apache.org 
>>> *Subject:* [EXTERNAL] [DISCUSS] NULL handling and the unfrozen
>>> collection issue
>>>
>>> You don't often get email from b.le...@gmail.com. Learn why this is
>>> important <https://aka.ms/LearnAboutSenderIdentification>
>>> Hi everybody,
>>>
>>> CEP-29 (CQL NOT Operator) is hitting the grey area of how we want as a
>>> community to handle NULL including for things like unfrozen (multi-cell)
>>> collections and I would like to make a proposal for moving forward with
>>> NULL related issues.
>>>
>>> We have currently 2 tickets open about NULL handling (I might have
>>> missed others):
>>>
>>>1. CASSANDRA-10715
>>><https://issues.apache.org/jira/browse/CASSANDRA-10715>: Allowing
>>>Filtering on NULL
>>>2. CASSANDRA-17762
>>>   

Re: [VOTE] Release Apache Cassandra Java Driver 4.18.1 (2nd attempt)

[Benjamin Lerer]

+1

Le jeu. 23 mai 2024, 21:30, Brandon Williams  a écrit :

> +1
>
> Kind Regards,
> Brandon
>
> On Tue, May 21, 2024 at 5:59 PM Bret McGuire 
> wrote:
> >
> >Greetings all!  We're going to give this another go.
> >
> >Apologies for the confusion that sprang out of our last attempt.  It
> appears that the Nexus staging repository for the 4.18.1 release was
> accidentally released shortly after it was created.  As a result Maven
> artifacts for this release are already out in the wild, so this vote will
> be a little unusual.  We'll be voting normally, but if the vote is
> successful we'll simply leave the Maven artifacts exactly where they are.
> If the vote is unsuccessful these artifacts will be removed from Maven
> Central and we'll try again with 4.18.2.
> >
> >Big thanks to mck and driftx for their help in untangling these
> issues.
> >
> >With all of that said let's get to it.  I’m proposing the test build
> of Cassandra Java Driver 4.18.1 for release.
> >
> > sha1: cbdde2878786fa6c4077a21352cbe738875f2106
> >
> > Git: https://github.com/apache/cassandra-java-driver/tree/4.18.1
> >
> > Maven Artifacts: As discussed above
> >
> >
> >The Source release and Binary convenience artifacts are available
> here:
> >
> >
> https://dist.apache.org/repos/dist/dev/cassandra/cassandra-java-driver/4.18.1/
> >
> >
> >The vote will be open for 72 hours (longer if needed). Everyone who
> has tested the build is invited to vote. Votes by PMC members are
> considered binding. A vote passes if there are at least three binding +1s
> and no -1's.
> >
> >One additional note: this is my first time doing a release of the
> Java driver so if you're so inclined you can find my PGP key information in
> the KEYS file.
> >
> >
> >Thanks all!
> >
> >
> >- Bret -
>

Re: [DISCUSS] Stream Pipelines on hot paths

[Benjamin Lerer]

For me the definition of hot path is too vague. We had arguments with
Berenger multiple times and it is more a waste of time than anything else
at the end. If we are truly concerned about stream efficiency then we
should simply forbid them. That will avoid lengthy discussions about what
constitute the hot path and what does not.

Le ven. 31 mai 2024 à 11:08, Berenguer Blasi  a
écrit :

> +1 on avoiding streams in hot paths
> On 31/5/24 9:48, Benedict wrote:
>
> My concept of hot path is simply anything we can expect to be called
> frequently enough in normal operation that it might show up in a profiler.
> If it’s a library method then it’s reasonable to assume it should be able
> to be used in a hot path unless clearly labelled otherwise.
>
> In my view this includes things that might normally be masked by caching
> but under supported workloads may not be - such as query preparation.
>
> In fact, I’d say the default assumption should probably be that a method
> is “in a hot path” unless there’s good argument they aren’t - such as that
> the operation is likely to be run at some low frequency and the slow part
> is not part of any loop. Repair setup messages perhaps aren’t a hot path
> for instance (unless many of them are sent per repair), but validation
> compaction or merkle tree construction definitely is.
>
> I think it’s fine to not have perfect agreement about edge cases, but if
> anyone in a discussion thinks something is a hot path then it should be
> treated as one IMO.
>
> On 30 May 2024, at 18:39, David Capwell 
>  wrote:
>
>  As a general statement I agree with you (same for String.format as
> well), but one thing to call out is that it can be hard to tell what is the
> hot path and what isn’t.  When you are doing background work (like repair)
> its clear, but when touching something internal it can be hard to tell;
> this can also be hard with shared code as it gets authored outside the hot
> path then later used in the hot path…
>
> Also, what defines hot path?  Is this user facing only?  What about
> Validation/Streaming (stuff processing a large dataset)?
>
> On May 30, 2024, at 9:29 AM, Benedict 
>  wrote:
>
> Since it’s related to the logging discussion we’re already having, I have
> seen stream pipelines showing up in a lot of traces recently. I am
> surprised; I thought it was understood that they shouldn’t be used on hot
> paths as they are not typically as efficient as old skool for-each
> constructions done sensibly, especially for small collections that may
> normally take zero or one items.
>
> I would like to propose forbidding the use of streams on hot paths without
> good justification that the cost:benefit is justified.
>
> It looks like it was nominally agreed two years ago that we would include
> words to this effect in the code style guide, but I forgot to include them
> when I transferred the new contents from the Google Doc proposal. So we
> could just include the “Performance” section that was meant to be included
> at the time.
>
> lists.apache.org
> 
> 
> 
> 
>
>
> On 30 May 2024, at 13:33, Štefan Miklošovič 
>  wrote:
>
> 
> I see the feedback is overall positive. I will merge that and I will
> improve the documentation on the website along with what Benedict suggested.
>
> On Thu, May 30, 2024 at 10:32 AM Mick Semb Wever  wrote:
>
>>
>>
>>
>>> Based on these findings, I went through the code and I have incorporated
>>> these rules and I rewrote it like this:
>>>
>>> 1) no wrapping in "if" if we are not logging more than 2 parameters.
>>> 2) rewritten log messages to not contain any string concatenation but
>>> moving it all to placeholders ({}).
>>> 3) wrap it in "if" if we need to execute a method(s) on parameter(s)
>>> which is resource-consuming.
>>>
>>
>>
>> +1
>>
>>
>> It's a shame slf4j botched it with lambdas, their 2.0 fluent api doesn't
>> impress me.
>>
>
>

[DISCUSS] LWT conditions behavior on collections is inconsistent (CASSANDRA-19637)

[Benjamin Lerer]

Hi everybody,

I wanted to raise attention to CASSANDRA-19637
 that I already
mentioned in the "[DISCUSS] NULL handling and the unfrozen collection
issue" thread.

The patch attempts to fix some inconsistencies in the way LWT conditions
work with collections when being compared to null values.

In case you feel that the solution is not going in the right direction or
you believe that there are some valid use cases that will be impacted by
this change in behavior feel free to raise the problem.

Re: [DISCUSS] Increments on non-existent rows in Accord

[Benjamin Lerer]

>
> Doesn’t an UPDATE statement creates a row if the partition key does not
> exist?


Things are more tricky that what the documentation is letting people
believe.

Internally, UPDATE does not really create a row. It creates a set of
updates for a row. At the CQL level it looks like a row exists but there is
a difference.
The best way to understand it is to run:
INSERT INTO my_table (pk, c, v) VALUES (1, 1, 1);
DELETE v FROM my_table WHERE pk = 1 AND c = 1;
SELECT * FROM my_table WHERE pk = 1 AND c =1;

The result will be:

 pk | c | v
+---+-
  1 | 1 | null


If instead you run

UPDATE my_table SET v = 1 WHERE pk = 1 AND c  = 1;
DELETE v FROM my_table WHERE pk = 1 AND c = 1;
SELECT * FROM my_table WHERE pk = 1 AND c =1;

The result will be:

 pk | c | v
+---+-

No row will be returned

An INSERT creates the primary key columns (by giving them a write
timestamp) and UPDATE does not.

Based on this behavior the second option appears to be counter
intuitive with how CQL normally operates. The first option looks to me
as the correct one as long as the user will know that the transaction
did not succeed.



Le ven. 21 juin 2024 à 17:05, Jon Haddad  a écrit :

> Seems to me that this should use the same behavior as a counter unless IF
> EXISTS is supplied.
>
> I can see a solid argument for failing though, if the argument is that
> only counters behave that way, vs increment / decrement.
>
>
>
> On Fri, Jun 21, 2024 at 4:32 PM Josh McKenzie 
> wrote:
>
>> What about aborting the transaction / raising an exception if you try and
>> do an incremental operator against a non-existent PK?
>>
>> The reasonable inferred intent of the user is to change a value that's
>> expected to be there, so if it's not there it's an error case right?
>> Otherwise you'd append "IF EXISTS".
>>
>> On Fri, Jun 21, 2024, at 1:56 AM, Caleb Rackliffe wrote:
>>
>> It does, but the primary reason it does is that it is setting a value,
>> not incrementing one. When we’re setting a value, we don’t care what was
>> there before. Incrementing a value is not possible in a non-transitional
>> update, hence this thread…
>>
>> On Jun 20, 2024, at 5:17 PM, Bernardo Botella <
>> conta...@bernardobotella.com> wrote:
>>
>> Doesn’t an UPDATE statement creates a row if the partition key does not
>> exist? That’s also confirmed by the official Cassandra documentation here
>> 
>> :
>>
>> ”Unlike in SQL, UPDATE does not check the prior existence of the row by
>> default. The row is created if none existed before, and updated otherwise.
>> Furthermore, there is no means of knowing which action occurred.”
>>
>> That being the case, I think the second option you mention is what keeps
>> consistency with the UPDATEs out of the transaction.
>>
>> Kind regards,
>> Bernardo
>>
>> On Jun 20, 2024, at 1:54 PM, Caleb Rackliffe 
>> wrote:
>>
>> We had a bug report a while back from Luis E Fernandez and team in
>> CASSANDRA-18988 
>> around the behavior of increments/decrements on numeric fields for
>> non-existent rows. Consider the following, wich can be run on the
>> cep-15-accord branch:
>>
>> CREATE KEYSPACE accord WITH replication = {'class': 'SimpleStrategy', 
>> 'replication_factor': '1'} AND durable_writes = true
>>
>>
>> CREATE TABLE accord.accounts (
>> partition text,
>> account_id int,
>> balance int,
>> PRIMARY KEY (partition, account_id)
>> ) WITH CLUSTERING ORDER BY (account_id ASC) AND transactional_mode='full'
>>
>>
>> BEGIN TRANSACTION
>> INSERT INTO accord.accounts (partition, account_id, balance) VALUES 
>> ('default', 0, 100);
>> INSERT INTO accord.accounts (partition, account_id, balance) VALUES 
>> ('default', 1, 100);
>> COMMIT TRANSACTION
>>
>>
>> BEGIN TRANSACTION
>> UPDATE accord.accounts SET balance -= 10 WHERE partition = 'default' AND 
>> account_id = 1;
>> UPDATE accord.accounts SET balance += 10 WHERE partition = 'default' AND 
>> account_id = 3;
>> COMMIT TRANSACTION
>>
>>
>> Reading the 'default' partition will produce the following result.
>>
>>
>>  partition | account_id | balance
>> ---++-
>>default |  0 | 100
>>default |  1 |  90
>>
>>
>> As you will notice, we have not implicitly inserted a row for account_id 3, 
>> which does not exist when we request that its balance be incremented by 10. 
>> This is by design, as null + 10 == null.
>>
>>
>> Before I close CASSANDRA-18988 
>> , *I'd like to 
>> confirm with everyone reading this that the behavior above is reasonable*. 
>> The only other option I've seen proposed that would make sense is perhaps 
>> producing a result like:
>>
>>
>>  partition | account_id | balance
>> ---++-
>>default |  0 | 100
>>default |  1 |  90
>>
>>   

Re: [VOTE][IP CLEARANCE] GoCQL driver

[Benjamin Lerer]

+1

Le mer. 26 juin 2024 à 08:39, Ekaterina Dimitrova  a
écrit :

> +1
>
> On Wed, 26 Jun 2024 at 6:14, Jon Haddad  wrote:
>
>> +1.
>>
>> On Wed, Jun 26, 2024 at 1:50 AM J. D. Jordan 
>> wrote:
>>
>>> +1 nb. Good to see this heavily used driver get continued development in
>>> the project.
>>>
>>> > On Jun 25, 2024, at 5:29 PM, Michael Shuler 
>>> wrote:
>>> >
>>> > +1
>>> >
>>> > Kind regards,
>>> > Michael
>>> >
>>> >> On 6/25/24 12:29, Mick Semb Wever wrote:
>>> >> Please vote on the acceptance of the GoCQL driver and its IP
>>> Clearance:
>>> >> https://incubator.apache.org/ip-clearance/cassandra-gocql-driver.html
>>> 
>>> >> All consent from original authors of the donation, and tracking of
>>> collected CLAs, is found in:
>>> >>  - https://github.com/gocql/gocql/issues/1751 <
>>> https://github.com/gocql/gocql/issues/1751>
>>> >>  -
>>> https://cwiki.apache.org/confluence/pages/worddav/preview.action?fileName=GoCQL+ASF+CLA+collection.xlsx&pageId=225152485
>>> <
>>> https://cwiki.apache.org/confluence/pages/worddav/preview.action?fileName=GoCQL+ASF+CLA+collection.xlsx&pageId=225152485
>>> >
>>> >> These do not require acknowledgement before thevote.
>>> >> The code is prepared for donation at https://github.com/gocql/gocql <
>>> https://github.com/gocql/gocql>
>>> >> Once thisvotepasses we will request ASF Infra to move the gocql/gocql
>>> as-is to apache/cassandra-gocql-driver  . The master branch and tags, with
>>> all their history, will be kept.  Because consent and CLAs were not
>>> received from all original authors the source files keep additional
>>> reference to their earlier copyright authors and license.
>>> >> This will become part of the Drivers subproject, ref CEP-8:
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+DataStax+Drivers+Donation
>>> <
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+DataStax+Drivers+Donation
>>> >
>>> >> PMC members, please check carefully the IP Clearance requirements
>>> beforevoting.
>>> >> Thevotewill be open for 72 hours (or longer).Votesby PMC members are
>>> considered binding. Avotepasses if there are at least three binding +1s and
>>> no -1's.
>>> >> regards,
>>> >> Mick
>>>
>>

Which approach should we use for exposing metrics through Virtual tables?

[Benjamin Lerer]

Hi,

I would like to start working on exposing the metrics through virtual
tables in CASSANDRA-14537


We had some long discussion already in CASSANDRA-7622 about which schema to
use to expose the metrics, unfortunately in the end I was not truly
convinced by any solution (including my own).

I would like to expose the possible solutions and there limitations and
advantages to find out which is the solution that people prefer or to see
if somebody can come up with another solution.

In CASSANDRA-7622, Chris Lohfink proposed to expose the table metric using
the following schema:

VIRTUAL TABLE table_stats (
keyspace_name TEXT,
table_name TEXT,
metric TEXT,
value DOUBLE,
fifteen_min_rate DOUBLE,
five_min_rate DOUBLE,
mean_rate DOUBLE,
one_min_rate DOUBLE,
p75th DOUBLE,
p95th DOUBLE,
p99th DOUBLE,
p999th DOUBLE,
min BIGINT,
max BIGINT,
mean DOUBLE,
std_dev DOUBLE,
median DOUBLE,
count BIGINT,
PRIMARY KEY( keyspace_name,  table_name , metric));

This approach has some advantages:

   - It is easy to use for all the metric categories that we have (http://
   cassandra.apache.org/doc/latest/operating/metrics.html)
   - The number of column is relatively small and fit in the cqlsh console.


The main disadvantage that I see with that approach is that it might not
always be super readable. Gauge or a Counter metric will have data for only
one column and will return NULL for all the others. If you know precisely
which metric is what and you only target that type of metric you can build
your query in such a way that the output is nicely formatted.
Unfortunately, I do not expect every user to know which metric is what.
The output format can also be problematic for monitoring tools as they
might have to use some extra logic to determine how to process each metric.

My preferred approach was to use metrics has columns. For example for the
threadpool metrics it will have given the following schema:

VIRTUAL TABLE threadpool_metrics (
pool_name TEXT,
active INT,
pending INT,
completed BIGINT,
blocked BIGINT,
total_blocked BIGINT,
max_pool_size INT,
PRIMARY KEY( pool_name )
)

That approach provide an output similar to the one of the nodetool
tpstats which will be, in my opinion, more readable that the previous
approach.

Unfortunately, it also has several serious drawbacks:


   - It does work for small set of metrics but do not work well for the
   table or keyspace metrics where we have more than 63 metrics. If you
   split the histograms, meters and timers into multiple columns you easily
   reach more than a hundred columns. As Chris pointed out in CASSANDRA-7622
   it makes the all thing unusable.
   - It also does not work properly for set of metrics like the commit log
   metrics because you can not get a natural primary key and will have to
   somehow create a fake one.


Nodetool solved the table and keyspace metric problems by splitting them
into subset (e.g. tablestats, tablehistograms). We could take a similar
approach and group metrics in meaningful sub-groups and expose them using
the second approach.

I tried to put myself in the shoes of a user that has a limited knowlegde
of the C* metrics but at the end of the day I am certainly not the best
person to figure out what is the best solution here. So I would like to
have your feedbacks on that problem.

Chris if I was wrong on some part or forgot some stuff feel free to correct
me.

Re: Would cqlsh describe command send query to cassandra server ?

[Benjamin Lerer]

Hi,

DESCRIBE commands are handled at the driver level not at the server level.
The drivers fetch metadata and keep them up to date.
When a DESCRIBE command is send, the driver build the output from those
metadata.

Benjamin

On Tue, Aug 7, 2018 at 1:33 PM, 陈仕（shichen）-技术产品中心  wrote:

>
> Hello, all.
>
> I try to use `desc keyspaces` or desc table t`, and at the same time
> capture network packets at Cassandra side with tcpdump.
> My tcpdump command is like this: tcpdump -i eth0 ip host 10.110.27.6 -X
> But I found no string such as desc or keyspaces or table in the output.
>
> So I wonder whether cqlsh fetch keyspace’s and table’s metadata just from
> local and not send query to server node ?
>
> Thanks.
>

Re: NEQ-restriction in select-where clause

[Benjamin Lerer]

It can be done for filtering, otherwise it is really tricky (I am not even
sure it can be done to be honest).



On Sun, Sep 9, 2018 at 2:11 AM, Dmitry Lazurkin  wrote:

> Hello.
>
> Does it make sense to implement != restriction in select-where clause?
> Is it possible?
>
> Thank you.
>
> PS. I want to implement that.
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Implicit Casts for Arithmetic Operators

[Benjamin Lerer]

Hi,

I would like to try to clarify things a bit to help people to understand
the true complexity of the problem.

The *float *and *double *types are inexact numeric types. Not only at the
operation level.

If you insert 676543.21 in a *float* column and then read it, you will
realize that the value has been truncated to 676543.2.

If you want accuracy the only way is to avoid those inexact types.
Using *decimals
*during operations will mitigate the problem but will not remove it.


I do not recall PostgreSQL behaving has described. If I am not mistaken in
PostgreSQL *SELECT 3/2* will return *1*. Which is similar to what MS SQL
server and Oracle do. So all thoses databases will lose precision if you
are not carefull.

If you truly need precision you can have it by using exact numeric types
for your data types. Of course it has a cost on performance, memory and
disk usage.

The advantage of the current approach is that it give you the choice. It is
up to you to decide what you need for your application. It is also in line
with the way CQL behave everywhere else.

Re: Implicit Casts for Arithmetic Operators

[Benjamin Lerer]

ays
> >> be
> >>>>> the
> >>>>>>>>>> return type.
> >>>>>>>>>>
> >>>>>>>>>> This would still leave a decision for float+double, though.  The
> >>> most
> >>>>>>>>>> consistent behaviour with that stated above would be to always
> >> take
> >>>>> the
> >>>>>>>>>> most approximate type to return (i.e. float), but this would
> seem
> >>> to
> >>>>> me
> >>>>>>>>>> to be fairly unexpected for the user.
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>>> On 12 Oct 2018, at 17:23, Ariel Weisberg 
> >>> wrote:
> >>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> I agree with what's been said about expectations regarding
> >>>>> expressions
> >>>>>>>> involving floating point numbers. I think that if one of the
> inputs
> >>> is
> >>>>>>>> approximate then the result should be approximate.
> >>>>>>>>>>>
> >>>>>>>>>>> One thing we could look at for inspiration is the SQL spec. Not
> >> to
> >>>>>>>> follow dogmatically necessarily.
> >>>>>>>>>>>
> >>>>>>>>>>> From the SQL 92 spec regarding assignment
> >>>>>>>>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__www.contrib.andrew.cmu.edu_-7Eshadow_sql_sql1992.txt&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=Jad7nE1Oab1mebx31r7AOfSsa0by8th6tCxpykmmOBA&m=vuYFCiEg1Hk9RcozkHxMcCqfg4quy5zdS6jn4LoxIog&s=2dMzYnFvO5Wf7J74IbDE27vxjfOX2xYT4-u7MEXUqHg&e=
> section
> >>> 4.6:
> >>>>>>>>>>> "
> >>>>>>>>>>>Values of the data types NUMERIC, DECIMAL, INTEGER,
> >> SMALLINT,
> >>>>>>>>>>>FLOAT, REAL, and DOUBLE PRECISION are numbers and are all
> >>>>>>>> mutually
> >>>>>>>>>>>comparable and mutually assignable. If an assignment would
> >>>>>>>> result
> >>>>>>>>>>>in a loss of the most significant digits, an exception
> >>>>> condition
> >>>>>>>>>>>is raised. If least significant digits are lost,
> >>>>> implementation-
> >>>>>>>>>>>defined rounding or truncating occurs with no exception
> >>>>>>>> condition
> >>>>>>>>>>>being raised. The rules for arithmetic are generally
> >> governed
> >>>>> by
> >>>>>>>>>>>Subclause 6.12, "".
> >>>>>>>>>>> "
> >>>>>>>>>>>
> >>>>>>>>>>> Section 6.12 numeric value expressions:
> >>>>>>>>>>> "
> >>>>>>>>>>>1) If the data type of both operands of a dyadic arithmetic
> >>>>>>>> opera-
> >>>>>>>>>>>   tor is exact numeric, then the data type of the result is
> >>>>>>>> exact
> >>>>>>>>>>>   numeric, with precision and scale determined as follows:
> >>>>>>>>>>> ...
> >>>>>>>>>>>2) If the data type of either operand of a dyadic arithmetic
> >>>>> op-
> >>>>>>>>>>>   erator is approximate numeric, then the data type of the
> >>> re-
> >>>>>>>>>>>   sult is approximate numeric. The precision of the result
> >> is
> >>>>>>>>>>>   implementation-defined.
> >>>>>>>>>>> "
> >>>>>>>>>>>
> >>>>>>>>>>> And this makes sense to me. I think we should only return an
> >> exact
> >>>>>>>> result if both of the inputs are exact.
> >>>>>>>>>>>
> >

Re: Implicit Casts for Arithmetic Operators

[Benjamin Lerer]

Then I would be interested in knowing `where we should be`. If the answer
is `ANSI SQL92` then my question is: Why? Simply for the sake of following
a standard?


On Thu, Nov 22, 2018 at 12:19 PM Benedict Elliott Smith 
wrote:

> As I say, for me this is explicitly unhelpful, so I have no intention of
> producing it (though, of course, I cannot prevent you from producing it)
>
> For me, the correct approach is to decide where we should be, and then
> figure out how to get there.  Where we are has no bearing on where we
> should be, in my view.
>
>
>
> > On 22 Nov 2018, at 11:12, Benjamin Lerer 
> wrote:
> >
> > I would also like to see an analysis of what being ANSI SQL 92 compliant
> > means in term of change of behavior (for arithmetics and *any features we
> > have already released*).
> > Simply because without it, I find the decision pretty hard to make.
> >
> > On Thu, Nov 22, 2018 at 11:51 AM Benedict Elliott Smith <
> bened...@apache.org <mailto:bened...@apache.org>>
> > wrote:
> >
> >> We’re not presently voting*; we’re only discussing, whether we should
> base
> >> our behaviour on a widely agreed upon standard.
> >>
> >> I think perhaps the nub of our disagreement is that, in my view, this is
> >> the only relevant fact to decide.  There is no data to base this
> decision
> >> upon.  It’s axiomatic, or ideological; procedural, not technical:  Do we
> >> think we should try to hew to standards (where appropriate), or do we
> think
> >> we should stick with what we arrived at in an adhoc manner?
> >>
> >> If we believe the former, as I now do, then the current state is only
> >> relevant when we come to implement the decision.
> >>
> >>
> >> * But given how peripheral and inherently ideological this decision is,
> >> and how meandering the discussion was with no clear consensus, it
> seemed to
> >> need a vote in the near future.  The prospect of a vote seems to have
> >> brought some healthy debate forward too, which is great, but I
> apologise if
> >> this somehow came across as presumptuous.
> >>
> >>
> >>> On 22 Nov 2018, at 09:26, Sylvain Lebresne  wrote:
> >>>
> >>> I'm not saying "let's not do this no matter what and ever fix technical
> >>> debt", nor am I fearing decision.
> >>>
> >>> But I *do* think decisions, technical ones at least, should be fact and
> >>> data driven. And I'm not even sure why we're talking of having a vote
> >> here.
> >>> The Apache Way is *not* meant to be primarily vote-driven, votes are
> >>> supposed to be a last resort when, after having debated facts and data,
> >> no
> >>> consensus can be reached. Can we have the debate on facts and data
> first?
> >>> Please.
> >>>
> >>> At the of the day, I object to: "There are still a number of unresolved
> >>> issues, but to make progress I wonder if it would first be helpful to
> >> have
> >>> a vote on ensuring we are ANSI SQL 92 compliant for our arithmetic?".
> >> More
> >>> specifically, I disagree that such vote is a good starting point. Let's
> >>> identify and discuss the unresolved issues first. Let's check precisely
> >>> what getting our arithmetic ANSI SQL 92 compliant means and how we can
> >> get
> >>> it. I do support the idea of making such analysis btw, it would be good
> >>> data, but no vote is needed whatsoever to make it. Again, I object to
> >>> voting first and doing the analysis 2nd.
> >>>
> >>> --
> >>> Sylvain
> >>>
> >>>
> >>> On Thu, Nov 22, 2018 at 1:25 AM Jonathan Haddad 
> >> wrote:
> >>>
> >>>> I can’t agree more. We should be able to make changes in a manner that
> >>>> improves the DB In the long term, rather than live with the technical
> >> debt
> >>>> of arbitrary decisions made by a handful of people.
> >>>>
> >>>> I also agree that putting a knob in place to let people migrate over
> is
> >> a
> >>>> reasonable decision.
> >>>>
> >>>> Jon
> >>>>
> >>>> On Wed, Nov 21, 2018 at 4:54 PM Benedict Elliott Smith <
> >>>> bened...@apache.org>
> >>>> wrote:
> >>>>
> >>>>> The goal is simply to agree on a set of well-d

Re: Implicit Casts for Arithmetic Operators

[Benjamin Lerer]

Sorry, following a standard for the sake of following a standard does not
make sense to me.

On Thu, Nov 22, 2018 at 12:33 PM Benedict Elliott Smith 
wrote:

> Yes.
>
> > On 22 Nov 2018, at 11:32, Benjamin Lerer 
> wrote:
> >
> > Then I would be interested in knowing `where we should be`. If the answer
> > is `ANSI SQL92` then my question is: Why? Simply for the sake of
> following
> > a standard?
> >
> >
> > On Thu, Nov 22, 2018 at 12:19 PM Benedict Elliott Smith <
> bened...@apache.org <mailto:bened...@apache.org>>
> > wrote:
> >
> >> As I say, for me this is explicitly unhelpful, so I have no intention of
> >> producing it (though, of course, I cannot prevent you from producing it)
> >>
> >> For me, the correct approach is to decide where we should be, and then
> >> figure out how to get there.  Where we are has no bearing on where we
> >> should be, in my view.
> >>
> >>
> >>
> >>> On 22 Nov 2018, at 11:12, Benjamin Lerer 
> >> wrote:
> >>>
> >>> I would also like to see an analysis of what being ANSI SQL 92
> compliant
> >>> means in term of change of behavior (for arithmetics and *any features
> we
> >>> have already released*).
> >>> Simply because without it, I find the decision pretty hard to make.
> >>>
> >>> On Thu, Nov 22, 2018 at 11:51 AM Benedict Elliott Smith <
> >> bened...@apache.org <mailto:bened...@apache.org>  bened...@apache.org <mailto:bened...@apache.org>>>
> >>> wrote:
> >>>
> >>>> We’re not presently voting*; we’re only discussing, whether we should
> >> base
> >>>> our behaviour on a widely agreed upon standard.
> >>>>
> >>>> I think perhaps the nub of our disagreement is that, in my view, this
> is
> >>>> the only relevant fact to decide.  There is no data to base this
> >> decision
> >>>> upon.  It’s axiomatic, or ideological; procedural, not technical:  Do
> we
> >>>> think we should try to hew to standards (where appropriate), or do we
> >> think
> >>>> we should stick with what we arrived at in an adhoc manner?
> >>>>
> >>>> If we believe the former, as I now do, then the current state is only
> >>>> relevant when we come to implement the decision.
> >>>>
> >>>>
> >>>> * But given how peripheral and inherently ideological this decision
> is,
> >>>> and how meandering the discussion was with no clear consensus, it
> >> seemed to
> >>>> need a vote in the near future.  The prospect of a vote seems to have
> >>>> brought some healthy debate forward too, which is great, but I
> >> apologise if
> >>>> this somehow came across as presumptuous.
> >>>>
> >>>>
> >>>>> On 22 Nov 2018, at 09:26, Sylvain Lebresne  <mailto:lebre...@gmail.com>> wrote:
> >>>>>
> >>>>> I'm not saying "let's not do this no matter what and ever fix
> technical
> >>>>> debt", nor am I fearing decision.
> >>>>>
> >>>>> But I *do* think decisions, technical ones at least, should be fact
> and
> >>>>> data driven. And I'm not even sure why we're talking of having a vote
> >>>> here.
> >>>>> The Apache Way is *not* meant to be primarily vote-driven, votes are
> >>>>> supposed to be a last resort when, after having debated facts and
> data,
> >>>> no
> >>>>> consensus can be reached. Can we have the debate on facts and data
> >> first?
> >>>>> Please.
> >>>>>
> >>>>> At the of the day, I object to: "There are still a number of
> unresolved
> >>>>> issues, but to make progress I wonder if it would first be helpful to
> >>>> have
> >>>>> a vote on ensuring we are ANSI SQL 92 compliant for our arithmetic?".
> >>>> More
> >>>>> specifically, I disagree that such vote is a good starting point.
> Let's
> >>>>> identify and discuss the unresolved issues first. Let's check
> precisely
> >>>>> what getting our arithmetic ANSI SQL 92 compliant means and how we
> can
> >>>> get
> >>>>> it. I do support the idea of making such analysis btw, it would be
>

Re: Warn about SASI usage and allow to disable them

[Benjamin Lerer]

 +1 on config. +1 on disabling by default

On Tue, Jan 15, 2019 at 6:51 AM Jeff Jirsa  wrote:

> +1 on config
> -0 on warning
> -0 on disabling by default
>
>
> --
> Jeff Jirsa
>
>
> > On Jan 14, 2019, at 9:22 PM, Taylor Cressy 
> wrote:
> >
> > +1 on config. +1 on disabling.
> >
> > +1 on applying it to materialized views as well.
> >
> >> On Jan 14, 2019, at 17:29, Joshua McKenzie 
> wrote:
> >>
> >> +1 on config change, +1 on disabling, and so long as the comments make
> the
> >> limitations and risks extremely clear, I'm fine w/out the client
> warning.
> >>
> >> On Mon, Jan 14, 2019 at 12:28 PM Andrés de la Peña <
> a.penya.gar...@gmail.com>
> >> wrote:
> >>
> >>> I mean disabling the creation of new SASI indices with CREATE INDEX
> >>> statement, the existing indexes would continue working. The CQL client
> >>> warning will be thrown with that creation statement as well (if they
> are
> >>> enabled).
> >>>
>  On Mon, 14 Jan 2019 at 20:18, Jeff Jirsa  wrote:
> 
>  When we say disable, do you mean disable creation of new SASI
> indices, or
>  disable using existing ones? I assume it's just creation of new?
> 
>  On Mon, Jan 14, 2019 at 11:19 AM Andrés de la Peña <
>  a.penya.gar...@gmail.com>
>  wrote:
> 
> > Hello all,
> >
> > It is my understanding that SASI is still to be considered an
> > experimental/beta feature, and they apparently are not being very
>  actively
> > developed. Some higlighted problems in SASI are:
> >
> > - OOMs during flush, as it is described in CASSANDRA-12662
> > - General secondary index consistency problems described in
>  CASSANDRA-8272.
> > There is a pending-review patch addressing the problem for regular
> 2i.
> > However, the proposed solution is based on indexing tombstones. SASI
> > doesn't index tombstones, so it wouldn't be enterely trivial to
> extend
>  the
> > approach to SASI.
> > - Probably insufficient testing. As far as I know, we don't have a
> >>> single
> > dtest for SASI nor tests dealing with large SSTables.
> >
> > Similarly to what CASSANDRA-13959 did with materialized views,
> > CASSANDRA-14866 aims to throw a native protocol warning about SASI
> > experimental state, and to add a config property to disable them.
> >>> Perhaps
> > this property could be disabled by default in trunk. This should
> raise
> > awareness about SASI maturity until we let them in a more stable
> state.
> >
> > The purpose for this thread is discussing whether we want to add this
> > warning, the config property and, more controversially, if we want to
> >>> set
> > SASI as disabled by default in trunk.
> >
> > WDYT?
> >
> 
> >>>
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Stability of MaterializedView in 3.11.x | 4.0

[Benjamin Lerer]

Hi Pankaj,

The main issues are described in the link you posted and are some
synchronization issues

* There's no way to determine if a view is out of sync with the base table.
* If you do determine that a view is out of sync, the only way to fix
it is to drop and rebuild
the view.
Even in the happy path, there isn’t an upper bound on how long it will
take for updates
to be reflected in the view.



On Tue, Aug 27, 2019 at 2:53 PM pankaj gajjar 
wrote:

> Hello,
>
>
>
> concern about Materialized Views (MVs) in Cassandra. Unfortunately starting
> with version 3.11, MVs are officially considered experimental and not ready
> for production use, as you can read here:
>
>
>
>
> https://urldefense.proofpoint.com/v2/url?u=http-3A__mail-2Darchives.apache.org_mod-5Fmbox_cassandra-2Duser_201710.mbox_-253CetPan.59f24f38.438f4e99.74dc-40apple.com-253E&d=DwIBaQ&c=adz96Xi0w1RHqtPMowiL2g&r=Jad7nE1Oab1mebx31r7AOfSsa0by8th6tCxpykmmOBA&m=gLWrMhkgn6VAhmUaYPdvhXIEHx0FOINcMtH1FxhC7i4&s=I4W1EfKR6JfWmm0x444DvOXyA4t3WpqsmBX20mlZphA&e=
>
>
>
> Can you please someone give some productive feedback on this ? it would
> help us to further implementation around the MVs in Cassandra.
>
>
>
> Does anyone facing some critical issue or data lose or synchronization
> issue ?
>
>
>
> Regards
>
> Pankaj.
>
> --
> --
> Regards
> Pankkaj.
>

Re: [proposal] Introduce AssertJ in test framework

[Benjamin Lerer]

+1

On Tue, Mar 10, 2020 at 6:18 PM Jon Haddad  wrote:

> I've used assertj in a lot of projects, I prefer it by a wide margin over
> using only junit.
>
> On Tue, Mar 10, 2020 at 9:45 AM David Capwell  wrote:
>
> > +1 from me
> >
> > In CASSANDRA-15564 I build my own assert chain to make the tests cleaner;
> > did it since assertj wasn't there.
> >
> > On Tue, Mar 10, 2020, 9:28 AM Kevin Gallardo <
> kevin.galla...@datastax.com>
> > wrote:
> >
> > > I would like to propose adding AssertJ <
> https://urldefense.proofpoint.com/v2/url?u=https-3A__assertj.github.io_doc_&d=DwIFaQ&c=adz96Xi0w1RHqtPMowiL2g&r=Jad7nE1Oab1mebx31r7AOfSsa0by8th6tCxpykmmOBA&m=WrkWi_LeOnAl7rqft1DM27OEkXD7sc2fnZMy_-c7IS8&s=D4FAaGRQi2WlwKAOQbQMfyt_cRqsOuZdePUDgchdLhA&e=
> >
> > as
> > > a test dependency and therefore have it available for writing
> > > unit/distributed/any test assertions.
> > >
> > > In addition to the examples mentioned on the AssertJ docs page (allows
> to
> > > do elaborate and comprehensible assertions on Collections, String, and
> > > *custom
> > > assertions*), here's an example of a dtest I was looking at, that could
> > be
> > > translated to AssertJ syntax, just to give an idea of how the syntax
> > would
> > > apply:
> > >
> > > *JUnit asserts*:
> > > try {
> > >[...]
> > > } catch (Exception e) {
> > > Assert.assertTrue(e instanceof RuntimeException);
> > > RuntimeException re = ((RuntimeException) e);
> > > Assert.assertTrue(re.getCause() instanceof ReadTimeoutException);
> > > ReadTimeoutException rte = ((ReadTimeoutException) e.getCause());
> > > Assert.assertTrue(rte.getMessage().contains("blabla")
> > >   && rte.getMessage().contains("andblablo"));
> > > }
> > >
> > > *AssertJ style:*
> > > try {
> > > [...]
> > > } catch (Exception e) {
> > > Assertions.assertThat(e)
> > > .isInstanceOf(RuntimeException.class)
> > > .hasCauseExactlyInstanceOf(ReadTimeoutException.class)
> > > .hasMessageContaining("blabla")
> > > .hasMessageContaining("andblablo");
> > > }
> > >
> > > The syntax is more explicit and more comprehensible, but more
> > importantly,
> > > when one of the JUnit assertTrue() fails, you don't know *why*, you
> only
> > > know that the resulting boolean expression is false.
> > > If a failure happened with the assertJ tests, the failure would say
> > > "Exception
> > > did not contain expected message, expected "blabla", actual
> "notblabla""
> > > (same for a lot of other situations), this makes debugging a failure,
> > after
> > > a test ran and failed much easier. With JUnit asserts you would have to
> > > additionally add a message explaining what the expected value is *and*
> > > what the
> > > actual value is, for each assert that is more complex than a
> assertEquals
> > > on a number, I suppose. I have seen a lot of tests so far that only
> test
> > > the expected behavior via assertTrue and does not show the incorrect
> > values
> > > when the test fails, which would come for free with AssertJ.
> > >
> > > Other examples randomly picked from the test suite:
> > >
> > >
> > >
> >
> *org.apache.cassandra.repair.RepairJobTest#testNoTreeRetainedAfterDistance:*
> > > Replace assertion:
> > > assertTrue(messages.stream().allMatch(m -> m.verb() == Verb.SYNC_REQ));
> > > With:
> > > assertThat(messages)
> > > .extracting(Message::verb)
> > > .containsOnly(Verb.SYNC_REQ);
> > >
> > > As a result, if any of the messages is not a Verb.SYNC_REQ, the test
> > > failure will show the actual "Verb"s of messages.
> > >
> > > Replace:
> > > assertTrue(millisUntilFreed < TEST_TIMEOUT_S * 1000);
> > > With:
> > > assertThat(millisUntilFreed)
> > > .isLessThan(TEST_TIMEOUT_S * 1000);
> > >
> > > Same effect if the condition is not satisfied, more explicit error
> > message
> > > explaining why the test failed.
> > >
> > > AssertJ also allows Custom assertions which are also very useful and
> > could
> > > potentially be leveraged in the future.
> > >
> > > This would only touch on the tests' assertions, the rest of the test
> > setup
> > > and execution remains untouched (still uses JUnit for the test
> > execution).
> > >
> > > Thanks.
> > >
> > > --
> > > Kévin Gallardo.
> > >
> >
>

Re: server side describe

[Benjamin Lerer]

+1 for shipping it after 4.0



On Thu, Apr 2, 2020 at 12:39 AM Jon Haddad  wrote:

> Most of the work to provide this feature is already done.  We need to
> generate server side CQL for snapshots already.  All we need to do is
> expose it via either a "DESCRIBE" CQL command, or I'm equally happy to see
> it land in a virtual table.
>
> On Wed, Apr 1, 2020 at 2:45 PM sankalp kohli 
> wrote:
>
> > +1 on holding off and focus on shipping 4.0
> >
> > On Wed, Apr 1, 2020 at 12:25 PM Joshua McKenzie 
> > wrote:
> >
> > > This looks like a feature that'd potentially invalidate some testing
> > that's
> > > been done and we've been feature frozen for over a year and a half.
> Also:
> > > scope creep.
> > >
> > > My PoV is we hold off. If we get into a cadence of more frequent
> releases
> > > we'll have it soon enough.
> > >
> > > On Wed, Apr 1, 2020 at 3:03 PM  wrote:
> > >
> > > > Hi,
> > > > Normally I ping the person on the ticket or in Slack to ask him/her
> for
> > > > status update and whether I can help. Then probably he/she gives me a
> > > > direction.
> > > > If I can’t find the person anymore, I just use my best judgement and
> > > > coordinate with people who might know better than me.
> > > > For now this strategy worked for me personally.
> > > > Hope this helps
> > > > Ekaterina
> > > >
> > > > Sent from my iPhone
> > > >
> > > > > On 1 Apr 2020, at 14:27, Jon Haddad  wrote:
> > > > >
> > > > > Hey folks,
> > > > >
> > > > > I was looking through our open JIRAs and realized we hadn't merged
> in
> > > > > server side describe calls yet.  The ticket died off a ways ago,
> and
> > I
> > > > > pinged Chris about it yesterday.  He's got a lot of his plate and
> > won't
> > > > be
> > > > > able to work on it anytime soon.  I still think we should include
> > this
> > > in
> > > > > 4.0.
> > > > >
> > > > > From a technical standpoint, It doesn't say much on the ticket
> after
> > > > Robert
> > > > > tossed an alternative patch out there.  I don't mind reviewing and
> > > > merging
> > > > > either of them, it sounded like both are pretty close to done and I
> > > think
> > > > > from the perspective of updating drivers for 4.0 this will save
> > quite a
> > > > bit
> > > > > of time since driver maintainers won't have to add new CQL
> generation
> > > for
> > > > > the various new options that have recently appeared.
> > > > >
> > > > > Questions:
> > > > >
> > > > > * Does anyone have an objection to getting this into 4.0? The
> patches
> > > > > aren't too huge, I think they're low risk, and also fairly high
> > reward.
> > > > > * I don't have an opinion (yet) on Robert's patch vs Chris's, with
> > > regard
> > > > > to which is preferable.
> > > > > * Since soon after Robert put up his PR he hasn't been around, at
> > least
> > > > as
> > > > > far as I've seen.  How have we dealt with abandoned patches before?
> > If
> > > > > we're going to add this in the patch will need some cleanup.  Is it
> > > > > reasonable to continue someone else's work when they've
> disappeared?
> > > > >
> > > > > Jon
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > > >
> > >
> >
>

Re: server side describe

[Benjamin Lerer]

It seems to me that we need to get better at making decisions for things
like that.
If we keep on arguing for small things, it will simply be time consuming
and painfull for everybody.
In this case, the situation seems simple.
Part of the group do not agree with the proposal. We just have to accept it
and move on.






On Fri, Apr 3, 2020 at 2:48 AM Joshua McKenzie  wrote:

> One thing probably worth thinking about: we're a mostly irascible lot to
> begin with and there's a global pandemic and Human Race Lockdown. I don't
> know about the rest of you but I'm starting from a pretty not-chill place
> these days; trying to be mindful of that.
>
> So for this: if we require a protocol change there's a clear forcing
> function to get it into 4.0 since a) it's minor and b) the time horizon for
> the next major is very uncertain.
>
> If we go a route that doesn't require a protocol change it can just go into
> the next patch release after 4.0 right? Is there an urgency to get it into
> 4.0.0 I'm missing in this scenario?
>
>
> On Thu, Apr 2, 2020 at 5:25 PM sankalp kohli 
> wrote:
>
> > Whether a feature is fully done and whether it validates or invalidate
> > testing is not the point here. The point is that it is a feature and
> > violates feature freeze. If someone brings in a feature which is almost
> > done and does not invalidate testing then will we merge all of them to
> 4.0?
> > Lot of features can be merged then based on this justification!!
> >
> >
> > Considering this is a small feature, I wont -1 on it.  I will disagree
> and
> > commit.
> >
> >
> >
> > On Thu, Apr 2, 2020 at 1:04 PM Jon Haddad  wrote:
> >
> > > Chris's original patch used a virtual table which didn't even require a
> > > protocol change.  To me, the difference between having a CQL describe
> vs
> > a
> > > virtual table is unimportant, since it's only drivers that need to care
> > > about it.  I'm completely fine with the simpler implementation of a
> > virtual
> > > table.
> > >
> > > Quite a bit of Chris's patch also fixes our broken server side CQL
> > > generation, something that affects correctness in our snapshots.  So
> > either
> > > way most of the code needs to go in before we release 4.0.  Adding a
> > single
> > > file that creates a new virtual table is so trivial I'm having a hard
> > time
> > > understanding the opposition.
> > >
> > > On Thu, Apr 2, 2020 at 12:56 PM Nate McCall 
> wrote:
> > >
> > > > So summarizing the salient points here:
> > > > - client authors have worked around this mostly, but this would avoid
> > > some
> > > > duplication of effort for new features
> > > > - this issues was tagged last year as being pertinent to 4.0 in
> several
> > > > threads about what was in scope
> > > > - there is some development efforts required to review/merge/update
> > these
> > > > patches and we are trying to release
> > > > - the change is unintrusive
> > > > - this is a change to the protocol
> > > >
> > > > Not having this doesnt effect me for $dayJob, but I want to reiterate
> > > that
> > > > it's a silly thing to leave to clients. Given we've previously scoped
> > it
> > > to
> > > > 4.0, im still +1 on adding it.
> > > >
> > > > It's ok to have differing opinions. I'd like to see us disagree and
> > > commit
> > > > to a course of action either way as opposed to just letting it sit
> more
> > > > because we can't sort it out.
> > > >
> > >
> >
>

Re: server side describe

[Benjamin Lerer]

We have already discussed it for several days and I believe that all the
points have been brought to the table.

I took the time to read CASSANDRA-14825 this morning to understand the full
story.
All the discussion has been about using Virtual Tables versus using
DESCRIBE.
My understanding is that a majority of people ended up in favor of a
DESCRIBE approach (skipping the details on purpose here).
Robert made a patch for that approach (according to his comment it was
discussed with Chris beforehand).

The question here is should: we agree to put it in 4.0 even if the version
is frozen for improvements.

My main reason against it is that if we broke the freeze for every ticket
we believe might be useful we will end up delaying 4.0 a lot.
Are there not other tickets that are more valuable than CASSANDRA-14825,
that are not included in 4.0?
Where do we draw the line?

I believe that if we were able to answer that question it would suddenly
become much easier to agree on which tickets we put in 4.0.








On Sun, Apr 5, 2020 at 1:25 AM Benedict Elliott Smith 
wrote:

> There it is.  I knew it would show up eventually.
>
> On 04/04/2020, 06:26, "bened...@apache.org" 
> wrote:
>
> > scope creep.
>
> I think it is unfair to label this scope creep; it would have to be
> newly considered for 4.0 for it to fall under that umbrella.
>
> I don't personally mind if it lands, but this was discussed at length
> on multiple occasions over the past year, and only stalled because of a
> combination of lack of etiquette, and a lack of leadership from e.g. PMC in
> resolving various political questions over the course of events.
>
> I also struggle to see how this would invalidate testing in any
> significant way?  It doesn't modify any existing behaviour.
>
> 
> From: Joshua McKenzie 
> Sent: 01 April 2020 19:24
> To: dev@cassandra.apache.org 
> Subject: Re: server side describe
>
> This looks like a feature that'd potentially invalidate some testing
> that's
> been done and we've been feature frozen for over a year and a half.
> Also:
> scope creep.
>
> My PoV is we hold off. If we get into a cadence of more frequent
> releases
> we'll have it soon enough.
>
> On Wed, Apr 1, 2020 at 3:03 PM  wrote:
>
> > Hi,
> > Normally I ping the person on the ticket or in Slack to ask him/her
> for
> > status update and whether I can help. Then probably he/she gives me a
> > direction.
> > If I can’t find the person anymore, I just use my best judgement and
> > coordinate with people who might know better than me.
> > For now this strategy worked for me personally.
> > Hope this helps
> > Ekaterina
> >
> > Sent from my iPhone
> >
> > > On 1 Apr 2020, at 14:27, Jon Haddad  wrote:
> > >
> > > Hey folks,
> > >
> > > I was looking through our open JIRAs and realized we hadn't merged
> in
> > > server side describe calls yet.  The ticket died off a ways ago,
> and I
> > > pinged Chris about it yesterday.  He's got a lot of his plate and
> won't
> > be
> > > able to work on it anytime soon.  I still think we should include
> this in
> > > 4.0.
> > >
> > > From a technical standpoint, It doesn't say much on the ticket
> after
> > Robert
> > > tossed an alternative patch out there.  I don't mind reviewing and
> > merging
> > > either of them, it sounded like both are pretty close to done and
> I think
> > > from the perspective of updating drivers for 4.0 this will save
> quite a
> > bit
> > > of time since driver maintainers won't have to add new CQL
> generation for
> > > the various new options that have recently appeared.
> > >
> > > Questions:
> > >
> > > * Does anyone have an objection to getting this into 4.0? The
> patches
> > > aren't too huge, I think they're low risk, and also fairly high
> reward.
> > > * I don't have an opinion (yet) on Robert's patch vs Chris's, with
> regard
> > > to which is preferable.
> > > * Since soon after Robert put up his PR he hasn't been around, at
> least
> > as
> > > far as I've seen.  How have we dealt with abandoned patches
> before?  If
> > > we're going to add this in the patch will need some cleanup.  Is it
> > > reasonable to continue someone else's work when they've
> disappeared?
> > >
> > > Jon
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
> >
>
>
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Keeping test-only changes out of CHANGES.txt

[Benjamin Lerer]

+1



On Thu, Apr 9, 2020 at 9:28 AM Eduard Tudenhoefner <
eduard.tudenhoef...@datastax.com> wrote:

> updated docs in https://github.com/apache/cassandra/pull/528
>
> On Wed, Apr 8, 2020 at 11:39 PM Jordan West  wrote:
>
> > +1 (nb) to the change and +1 (nb) to updating the docs to reflect this.
> >
> > Jordan
> >
> > On Wed, Apr 8, 2020 at 11:30 AM  wrote:
> >
> > > +1
> > >
> > > > El 8 abr 2020, a las 19:05, e.dimitr...@gmail.com escribió:
> > > >
> > > > +1
> > > >
> > > > Sent from my iPhone
> > > >
> > > >> On 8 Apr 2020, at 13:50, Joshua McKenzie 
> > wrote:
> > > >>
> > > >> +1
> > > >>
> > >  On Wed, Apr 8, 2020 at 12:26 PM Sam Tunnicliffe 
> > > wrote:
> > > >>>
> > > >>> +1
> > > >>>
> > > > On 8 Apr 2020, at 15:08, Mick Semb Wever  wrote:
> > > 
> > >  Can we agree on keeping such test changes out of CHANGES.txt ?
> > > 
> > >  We already don't put entries into CHANGES.txt if it is not a
> change
> > >  from any previous release.
> > > 
> > >  There was some discussion before¹ about this, and the problem that
> > >  being selective meant what ended up there being arbitrary. I think
> > >  this can be solved with an easy rule of thumb that if it only
> > touches
> > >  *Test.java classes, or it is only about fixing a test, then it
> > >  shouldn't be in CHANGES.txt. That means if the patch does touch
> any
> > >  runtime code then you do still need to add an entry to
> CHANGES.txt.
> > >  This avoids the whole "arbitrary" problem,  and maintains
> > CHANGES.txt
> > >  as user-facing formatted text to be searched through.
> > > 
> > >  If there's agreement I can commit to going through 4.0 changes and
> > >  removing those that never touched runtime code.
> > > 
> > >  regards,
> > >  Mick
> > > 
> > >  ¹)
> > > >>>
> > >
> >
> https://lists.apache.org/thread.html/a94946887081d8a408dd5cd01a203664f4d0197df713f0c63364a811%40%3Cdev.cassandra.apache.org%3E
> > > 
> > > 
> > -
> > >  To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > >  For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > 
> > > >>>
> > > >>>
> > > >>>
> -
> > > >>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > >>> For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >>>
> > > >>>
> > > >
> > > > -
> > > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > > >
> > >
> > > -
> > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > > For additional commands, e-mail: dev-h...@cassandra.apache.org
> > >
> > >
> >
>
>
> --
> Eduard Tudenhoefner
> e. eduard.tudenhoef...@datastax.com
> w. www.datastax.com
>

Re: server side describe

[Benjamin Lerer]

Thanks for your answer Alex.

Some of the things we
> have explicitly (as a community) agreed to commit are still in progress,
> including several client protocol changes. And, to my understanding, if
> those are committed, it'll be what community has agreed upon.
>

If the understanding of the community was that this patch was expected to
go in 4.0 then it makes sense to commit it there.


> If the discussion is about whether or not to include _some_ version of the
> patch, I think including it makes sense, especially given the patch was
> there for a while and was not committed for non-technical reasons. If we're
> trying to decide _which_ patch to commit, I'd personally focus on the
> original patch (to foster recognition), get it reviewed, and pick any
> changes that make it better from the follow-up one (to foster
> collaboration).
>

I am fine going through both patches and figuring out a way to merge them
into one if Jon or somebody else is willing to review the output of that
merge



On Thu, Apr 9, 2020 at 12:45 PM Benedict Elliott Smith 
wrote:

> +1 all of this
>
> On 09/04/2020, 10:23, "Oleksandr Petrov" 
> wrote:
>
> > My understanding is that a majority of people ended up in favor of
> a DESCRIBE approach. Robert made a patch for that approach (according
> to
> his comment it was discussed with Chris beforehand).
>
> This sounds like just a switch of API (in other words, you use the same
> string generation, but return it via different means). From the
> (little)
> time I've spent looking at the patch, I remember that there wasn't much
> common code between the two (sorry if I'm remembering it wrong).
>
> If the discussion is about whether or not to include _some_ version of
> the
> patch, I think including it makes sense, especially given the patch was
> there for a while and was not committed for non-technical reasons. If
> we're
> trying to decide _which_ patch to commit, I'd personally focus on the
> original patch (to foster recognition), get it reviewed, and pick any
> changes that make it better from the follow-up one (to foster
> collaboration).
>
> > Where do we draw the line?
>
> I would discuss changes on the case-by-case basis. Some of the things
> we
> have explicitly (as a community) agreed to commit are still in
> progress,
> including several client protocol changes. And, to my understanding, if
> those are committed, it'll be what community has agreed upon. All
> further
> changes that weren't discussed yet - we can talk over. If it's
> something as
> trivial as server-side describe that doesn't risk stability but adds a
> lot
> of value, it may make sense to include. But I woudln't attempt to come
> up
> with a general rule for what we may or may not consider for now.
>
>
> On Mon, Apr 6, 2020 at 3:21 PM Benjamin Lerer <
> benjamin.le...@datastax.com>
> wrote:
>
> > We have already discussed it for several days and I believe that all
> the
> > points have been brought to the table.
> >
> > I took the time to read CASSANDRA-14825 this morning to understand
> the full
> > story.
> > All the discussion has been about using Virtual Tables versus using
> > DESCRIBE.
> > My understanding is that a majority of people ended up in favor of a
> > DESCRIBE approach (skipping the details on purpose here).
> > Robert made a patch for that approach (according to his comment it
> was
> > discussed with Chris beforehand).
> >
> > The question here is should: we agree to put it in 4.0 even if the
> version
> > is frozen for improvements.
> >
> > My main reason against it is that if we broke the freeze for every
> ticket
> > we believe might be useful we will end up delaying 4.0 a lot.
> > Are there not other tickets that are more valuable than
> CASSANDRA-14825,
> > that are not included in 4.0?
> > Where do we draw the line?
> >
> > I believe that if we were able to answer that question it would
> suddenly
> > become much easier to agree on which tickets we put in 4.0.
> >
> >
> >
> >
> >
> >
> >
> >
> > On Sun, Apr 5, 2020 at 1:25 AM Benedict Elliott Smith <
> bened...@apache.org
> > >
> > wrote:
> >
> > > There it is.  I knew it would show up eventually.
> > >
> > > On 04/04/2020, 06:26, "bened...

Re: [VOTE] Release Apache Cassandra 4.0-alpha4

[Benjamin Lerer]

+1



On Tue, Apr 14, 2020 at 5:50 PM Blake Eggleston
 wrote:

> +1
>
> > On Apr 14, 2020, at 5:09 AM, e.dimitr...@gmail.com wrote:
> >
> > I also can’t see them. I think it matters to which interface is the
> link.
> >
> > And +1 from me, thanks!
> >
> >> On 14 Apr 2020, at 7:53, Erick Ramirez 
> wrote:
> >>
> >> 
> >>>
> >>>
>  All java8 UTs, jvmdtests and dtests pass
> 
> >>>
> https://circleci.com/workflow-run/d7b3f62d-c9ad-43d6-9152-2655e27feccb?signup-404=true
> >>>
> >>>
> >>> Is anyone else able to see this^ circleci page?
> >>> For me, it never loads, and isn't the first time I've been unable to
> >>> see others' circleci results.
> >>>
> >>
> >> All that shows up for me is:
> >>
> >> Workflows  >>  null  >>  null  >> null
> >> 0 jobs in this workflow
> >>
> >>
> >> and a spinning widget.
> >
> > -
> > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> > For additional commands, e-mail: dev-h...@cassandra.apache.org
> >
>
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: server side describe

[Benjamin Lerer]

Now that there is an agreement on having this patch on 4.0. I would like to
see it done as soon as possible. It would minimize the risks by giving us
more testing time. It will also give the driver developers a chance to use
it and provide feedback if they hit some issues.

Robert rebased his patch yesterday and I started to review it. If somebody
else want to give another look to the patch, I am 100% for it.




On Wed, Apr 15, 2020 at 7:48 PM Jon Haddad  wrote:

> I don't think we need to vote on this.  A few folks have expressed (very
> justified) concern, but nobody has gone nuclear.
>
> I intended on cleaning up the patch and getting it ready for another
> review, but haven't had it in my brain and have been busy with other
> things.  I don't claim exclusive rights to the clean up patch, if someone
> was motivated to do it I'd be 100% fine with it. :)
>
>
>
>
> On Wed, Apr 15, 2020 at 10:15 AM sankalp kohli 
> wrote:
>
> > What are the next steps? Do we hold a vote as I see few initial emails
> > which dont seem to agree and have not replied based on future
> discussions.
> >
> > On Fri, Apr 10, 2020 at 12:50 PM Dinesh Joshi  wrote:
> >
> > > +1 let's keep moving forward.
> > >
> > > Dinesh
> > >
> > > > On Apr 9, 2020, at 4:07 PM, Nate McCall  wrote:
> > > >
> > > > Ok cool. It sounds like we are moving this towards a consensus? (at
> > least
> > > > agreeing to disagree and moving forward).
> > > >
> > > > I very much the different inputs on this thread - thanks for a
> largely
> > > > healthy discussion folks.
> > > >
> > > > On Fri, Apr 10, 2020 at 6:03 AM Chris Lohfink 
> > > wrote:
> > > >
> > > >> I'd be in favor of going with the newer DESCRIBE option. The
> original
> > > patch
> > > >> was mostly focused on just getting the CQL correct and used virtual
> > > tables
> > > >> because its what the initial feedback was to do. Robert added a lot
> of
> > > >> functionality on top of what was there which is what people were
> > > starting
> > > >> to ask for. I am in favor of just going off of his patch and moving
> > > >> forward. I initially heard post 4.0 so I know I haven't been focused
> > on
> > > it
> > > >> but if theres desire to put it in 4.0 I am +1 on it and would like
> to
> > > see
> > > >> just going off of the Robert's latest patch.
> > > >>
> > > >> Chris
> > > >>
> > > >> On Thu, Apr 9, 2020 at 6:41 AM Benjamin Lerer <
> > > benjamin.le...@datastax.com
> > > >>>
> > > >> wrote:
> > > >>
> > > >>> Thanks for your answer Alex.
> > > >>>
> > > >>> Some of the things we
> > > >>>> have explicitly (as a community) agreed to commit are still in
> > > >> progress,
> > > >>>> including several client protocol changes. And, to my
> understanding,
> > > if
> > > >>>> those are committed, it'll be what community has agreed upon.
> > > >>>>
> > > >>>
> > > >>> If the understanding of the community was that this patch was
> > expected
> > > to
> > > >>> go in 4.0 then it makes sense to commit it there.
> > > >>>
> > > >>>
> > > >>>> If the discussion is about whether or not to include _some_
> version
> > of
> > > >>> the
> > > >>>> patch, I think including it makes sense, especially given the
> patch
> > > was
> > > >>>> there for a while and was not committed for non-technical reasons.
> > If
> > > >>> we're
> > > >>>> trying to decide _which_ patch to commit, I'd personally focus on
> > the
> > > >>>> original patch (to foster recognition), get it reviewed, and pick
> > any
> > > >>>> changes that make it better from the follow-up one (to foster
> > > >>>> collaboration).
> > > >>>>
> > > >>>
> > > >>> I am fine going through both patches and figuring out a way to
> merge
> > > them
> > > >>> into one if Jon or somebody else is willing to review the output of
> > > that
> > > >>> merge
> > >