Re: [DISCUSS] Donating easy-cass-stress to the project

[David Capwell]

> So: besides Jon, who in the community expects/desires to maintain this going 
> forward? 

I have been maintaining a fork for years, so don’t mind helping maintain this 
project.

> On Apr 28, 2024, at 4:08 AM, Mick Semb Wever  wrote:
> 
>> A separate subproject like dtest and the Java driver would maybe help 
>> address concerns with introducing a gradle build system and Kotlin.
> 
> 
> 
> Nit, dtest is a separate repository, not a subproject.  The Java driver is 
> one repository to be in the Drivers subproject.  Esoteric maybe, but ASF 
> terminology we need to get right :-) 
> 
> To your actual point (IIUC), it can be a separate repository and not a 
> separate subproject.  This permits it to be kotlin+gradle, while not having 
> the formal subproject procedures.  It still needs 3 responsible committers 
> from the get-go to show sustainability.  Would easy-cass-stress have 
> releases, or always be a codebase users work directly with ?
> 
> Can/Should we first demote cassandra-stress by moving it out to a separate 
> repo ? 
>  ( Can its imports work off non-snapshot dependencies ? )
> It might feel like an extra prerequisite step to introduce, but maybe it 
> helps move the needle forward and make this conversation a bit easier/obvious.
> 

Re: [DISCUSS] Fixing coverage reports for jvm-dtest-upgrade

[David Capwell]

Thanks, looks good to me!

> On Mar 19, 2024, at 10:19 AM, Abe Ratnofsky  wrote:
> 
> Here's where ClassLoader names were introduced in JDK9: 
> https://bugs.java.com/bugdatabase/view_bug?bug_id=6479237
> 
> And where Jacoco introduced support for the exclclassloader agent option: 
> https://github.com/jacoco/jacoco/commit/b8ee4efe9c2ba93485fe5d9e25340113efc2390b
> 
> My understanding is that names only exist to improve error messages 
> (ClassCastException specifically), and other diagnostic features like 
> Jacoco's exclusions. I am not aware of any behavior-impacting reasons to name 
> a ClassLoader, or anything this would break by adding ClassLoader names, but 
> I'm testing my changes now and will report back. Naming is done just by 
> overriding ClassLoader.getName: 
> https://docs.oracle.com/javase/9/docs/api/java/lang/ClassLoader.html#getName--
> 
>> On Mar 18, 2024, at 3:48 PM, David Capwell  wrote:
>> 
>> Are there any side effects to naming ClassLoaders?  How do we do this?
>> 
>> I am +1 to the idea but don’t know enough to know what negatives could 
>> happen.
>> 
>>> On Mar 17, 2024, at 7:25 AM, Josh McKenzie  wrote:
>>> 
>>> +1 from me
>>> 
>>>> If classloaders are appropriately named in the current versions of 
>>>> Cassandra, we should be able to test upgrade paths to that version without 
>>>> updating the older branches or building new jvm-dtest JARs for them.
>>> Pretty sure Caleb was wrestling w/something recently that might have 
>>> benefited from being able to differentiate which ClassLoader was sad; in 
>>> general this seems like it'd be a big help to debugging startup / env 
>>> issues, assuming it doesn't break anything. :)
>>> 
>>> On Fri, Mar 15, 2024, at 4:41 PM, Abe Ratnofsky wrote:
>>>> Hey folks,
>>>> 
>>>> I'm working on gathering coverage data across all of our test suites. The 
>>>> jvm-dtest-upgrade suite is currently unfriendly to Jacoco: it uses classes 
>>>> from multiple versions of Cassandra but with the same class names, so 
>>>> Jacoco's analysis fails due to "Can't add different class with same 
>>>> name"[1]. We need a way to exclude alternate-version classes from Jacoco's 
>>>> analysis, so we can get coverage for the current version of Cassandra.
>>>> 
>>>> Jacoco supports exclusion of classes based on class name or classloader 
>>>> name, but the class names are frequently usually identical across 
>>>> Cassandra versions. The jvm-dtest framework doesn't name classloaders, so 
>>>> we can't use that either.
>>>> 
>>>> I'd like to propose that we name the jvm-dtest InstanceClassLoader 
>>>> instances so that some can be excluded from Jacoco's analysis. Instances 
>>>> that create new InstanceClassLoaders should be able to provide an 
>>>> immutable name in the constructor. InstanceClassLoader names should 
>>>> indicate whether they're for the current version of Cassandra (where 
>>>> coverage should be collected) or an alternate version. If classloaders are 
>>>> appropriately named in the current versions of Cassandra, we should be 
>>>> able to test upgrade paths to that version without updating the older 
>>>> branches or building new jvm-dtest JARs for them.
>>>> 
>>>> Any objections or alternate approaches?
>>>> 
>>>> --
>>>> Abe
>>>> 
>>>> [1]: More specifically: Jacoco uses class IDs to map the analysis data 
>>>> that's produced during text execution to .class files. I'm configuring the 
>>>> Jacoco agent's classdumpdir to ensure that the classes loaded during 
>>>> execution are the same classes that are analyzed during report generation, 
>>>> as is recommended. When we build the alternate version JARs for 
>>>> jvm-dtest-upgrade, we end up with multiple classes with the same name but 
>>>> different IDs.
>> 
> 

Re: [DISCUSS] Fixing coverage reports for jvm-dtest-upgrade

[David Capwell]

Are there any side effects to naming ClassLoaders?  How do we do this?

I am +1 to the idea but don’t know enough to know what negatives could happen.

> On Mar 17, 2024, at 7:25 AM, Josh McKenzie  wrote:
> 
> +1 from me
> 
>> If classloaders are appropriately named in the current versions of 
>> Cassandra, we should be able to test upgrade paths to that version without 
>> updating the older branches or building new jvm-dtest JARs for them.
> Pretty sure Caleb was wrestling w/something recently that might have 
> benefited from being able to differentiate which ClassLoader was sad; in 
> general this seems like it'd be a big help to debugging startup / env issues, 
> assuming it doesn't break anything. :)
> 
> On Fri, Mar 15, 2024, at 4:41 PM, Abe Ratnofsky wrote:
>> Hey folks,
>> 
>> I'm working on gathering coverage data across all of our test suites. The 
>> jvm-dtest-upgrade suite is currently unfriendly to Jacoco: it uses classes 
>> from multiple versions of Cassandra but with the same class names, so 
>> Jacoco's analysis fails due to "Can't add different class with same 
>> name"[1]. We need a way to exclude alternate-version classes from Jacoco's 
>> analysis, so we can get coverage for the current version of Cassandra.
>> 
>> Jacoco supports exclusion of classes based on class name or classloader 
>> name, but the class names are frequently usually identical across Cassandra 
>> versions. The jvm-dtest framework doesn't name classloaders, so we can't use 
>> that either.
>> 
>> I'd like to propose that we name the jvm-dtest InstanceClassLoader instances 
>> so that some can be excluded from Jacoco's analysis. Instances that create 
>> new InstanceClassLoaders should be able to provide an immutable name in the 
>> constructor. InstanceClassLoader names should indicate whether they're for 
>> the current version of Cassandra (where coverage should be collected) or an 
>> alternate version. If classloaders are appropriately named in the current 
>> versions of Cassandra, we should be able to test upgrade paths to that 
>> version without updating the older branches or building new jvm-dtest JARs 
>> for them.
>> 
>> Any objections or alternate approaches?
>> 
>> --
>> Abe
>> 
>> [1]: More specifically: Jacoco uses class IDs to map the analysis data 
>> that's produced during text execution to .class files. I'm configuring the 
>> Jacoco agent's classdumpdir to ensure that the classes loaded during 
>> execution are the same classes that are analyzed during report generation, 
>> as is recommended. When we build the alternate version JARs for 
>> jvm-dtest-upgrade, we end up with multiple classes with the same name but 
>> different IDs.

Re: [DISCUSSION] Replace the Config class instance with the tree-based framework

[David Capwell]

Just went over the doc and I don’t see a real argument for why we want to 
switch “frameworks”… I think this should be fleshed out more; what is it we are 
actually solving for?

Also, I “think” its trying to propose we switch from a mostly flat yaml to a 
nested structure… I kinda feel this should be separated as we can and do do 
this today but the reason we have not grouped is not due to the framework we 
use but more “what makes sense to group” (lack of agreement and consistency 
here… ).  There was a JIRA that Caleb worked on and him and Benedict came up 
with a proposal, I recommend taking a look at that past work if you wish to do 
something similar.

We all have to maintain our configs, so we need to be able to answer a few 
questions with this type of proposal

1) how will developers maintain, and does this add any burdens to them?  What 
benefits do I get as a maintainer?
2) what are the read costs.  During a single CQL command we do many accesses to 
Config to see what we should and should not do/allow, so the costs here must be 
known or easy to understand.
3) config is a critical and core part of the code, so a rewrite comes with huge 
risk and cost, do we have equal or greater reward for this?  What are those 
rewards?  What are those risks?
4) can the motivations for this work be solved in a different way that improves 
one or more of the above questions?  Why is your proposal more desirable than 
those solutions?

> On Mar 13, 2024, at 12:15 PM, Maxim Muzafarov  wrote:
> 
> Hello everyone,
> 
> During the implementation, many valid design comments were made about
> making the virtual table SettingTable [1] updatable. So, I've
> rethought the whole concept once again, and I want to take another
> step forward to make this problem feasible with less effort on our
> part.
> 
> I want to replace the way we use and store node configuration values
> internally, which means I want to replace the Config class instance,
> where we store values with a tree-based framework.
>>> I propose to use the Lightbend API to do this. <<
> 
> The changes themselves are quite limited, they don't require rewriting
> the whole codebase. All the DatabaseDescriptor methods will be
> retained, and the only thing that would change is the way we store the
> values (in the in-memory tree, not in the Config class instance
> itself). So I don't expect that it will be a huge change.
> 
> 
> All the design details are in the document below, including the
> framework comparison, the API, and the way how we will manage the
> configuration schema.
> 
> Please take a look, I want to move things forward as every important
> change pulls on a bigger problem that has been neglected for years :-)
> Let's agree on the framework/API we want to use so that I can invest
> more time in the implementation.
> 
> https://docs.google.com/document/d/11W1Qj-6d9ZqHv86iEKgFutcxY2DTMIofEbr-zQiw930/edit#heading=h.2051pbez4rce
> 
> Looking forward to your comments.
> 
> [1] https://issues.apache.org/jira/browse/CASSANDRA-15254

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[David Capwell]

> Is it not possible to install and run python 3.8 on RHEL 7?


You have a few options, none really good.

1) build from source
2) a RPM from outside of RHEL; this means you don’t have support and must trust 
a different third party (not managed by python or RedHat)
3) you use SCL which means every time you want to touch CQLSH you have to 
remember to enable it (as its per-bash session)

These 3 options don’t really work for most deployments


> On Mar 11, 2024, at 1:12 PM, Brad  wrote:
> 
> RHEL 7 will reach the end of maintenance on June 30th, 2024 (extended 
> lifecycle support is an option).
> 
> Is it not possible to install and run python 3.8 on RHEL 7?  I assume that 
> would be necessary to run Java 11 on RHEL 7 with Cassandra 5.0.  It would be 
> a burden for contributors to test with an obsolete version of python -- you 
> can't 'brew install python@3.6' for example.
> 
> % brew install python@3.6
> Warning: No available formula with the name "python@3.6"
> 
> % brew install python@3.7 
> Error: python@3.7 has been disabled because it is deprecated upstream!
> 
> On Mon, Mar 11, 2024 at 3:38 PM Caleb Rackliffe  <mailto:calebrackli...@gmail.com>> wrote:
>> I can try this out on trunk. Will report back...
>> 
>> On Mon, Mar 11, 2024 at 2:23 PM J. D. Jordan > <mailto:jeremiah.jor...@gmail.com>> wrote:
>>> The Python driver dropped official support for older EOL Python versions 
>>> because they are EOL and no longer tested by the newer driver CI. I don’t 
>>> think there are actually any changes yet that it won’t work in 3.6 still?  
>>> Maybe someone with Python 3.6 installed can change the if and see?  I think 
>>> we have some cqlsh tests in dtest?  As long as we as a project run those on 
>>> RHEL 7, I would be comfortable with adding that back to being supported. 
>>> Though maybe just in the rpm package?
>>> 
>>> -Jeremiah
>>> 
>>>> On Mar 11, 2024, at 1:33 PM, Josh McKenzie >>> <mailto:jmcken...@apache.org>> wrote:
>>>> 
>>>> 
>>>> Looks like we bumped from 3.6 requirement to 3.7 in CASSANDRA-18960 
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18960> as well - similar 
>>>> thing. Vector support in python, though that patch took it from "return a 
>>>> simple blob" to "return something the python driver knows about, but 
>>>> apparently not variable types so we'll need to upgrade again."
>>>> 
>>>>> The version of the Python driver that is used by cqlsh (3.25.0) doesn't 
>>>>> entirely support the new vector data type introduced by CASSANDRA-18504 
>>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18504>. While we can 
>>>>> perfectly write data, read vectors are presented as blobs:
>>>> 
>>>> As far as I can tell, support for vector types in cqlsh is the sole reason 
>>>> we've bumped to 3.7 and 3.8 to support that python driver. That correct 
>>>> Andres / Brandon?
>>>> 
>>>> On Mon, Mar 11, 2024, at 1:22 PM, Caleb Rackliffe wrote:
>>>>> The vector issues itself was a simple error message change: 
>>>>> https://github.com/datastax/python-driver/commit/e90c0f5d71f4cac94ed80ed72c8789c0818e11d0
>>>>> 
>>>>> Was there something else in 3.29.0 that actually necessitated the move to 
>>>>> a floor of Python 3.8? Do we generally change runtime requirements in 
>>>>> minor releases for the driver?
>>>>> 
>>>>> On Mon, Mar 11, 2024 at 12:12 PM Brandon Williams >>>> <mailto:dri...@gmail.com>> wrote:
>>>>> Given that 3.6 has been EOL for 2+ years[1], I don't think it makes
>>>>> sense to add support for it back.
>>>>> 
>>>>> Kind Regards,
>>>>> Brandon
>>>>> 
>>>>> [1] https://devguide.python.org/versions/
>>>>> 
>>>>> On Mon, Mar 11, 2024 at 12:08 PM David Capwell >>>> <mailto:dcapw...@apple.com>> wrote:
>>>>> >
>>>>> > Originally we had planned to support RHEL 7 but in testing 5.0 we found 
>>>>> > out that cqlsh no longer works on RHEL 7[1].  This was changed in 
>>>>> > CASSANDRA-19245 which upgraded python-driver from 3.28.0 to 3.29.0. For 
>>>>> > some reason this minor version upgrade also dropped support for python 
>>>>> > 3.6 which is the supported python version on RHEL 7.
>>>>> >
>>>>> > We wanted to bring this to the attention of the community to figure out 
>>>>> > next steps; do we wish to say that RHEL 7 is no longer supported 
>>>>> > (making upgrades tied to OS upgrades, which can be very hard for 
>>>>> > users), or do we want to add python 3.6 support back to python-driver?
>>>>> >
>>>>> >
>>>>> > 1: the error seen by users is
>>>>> > $ cqlsh
>>>>> > Warning: unsupported version of Python, required 3.8-3.11 but found 3.6 
>>>>> > Warning: unsupported version of Python, required 3.8-3.11 but found 2.7
>>>>> > No appropriate Python interpreter found.
>>>>> > $
>>>>> >
>>>>> >
>>>> 

Re: [DISCUSS] Cassandra 5.0 support for RHEL 7

[David Capwell]

> I don't think it makes sense to add support for it back.

Was there a discussion thread about this?  My understanding is all breaking 
changes must have a discussion thread and library changes like this would as 
well.  Very possible I missed it, so wanted to read that thread to see any 
history here.

> On Mar 11, 2024, at 12:22 PM, J. D. Jordan  wrote:
> 
> The Python driver dropped official support for older EOL Python versions 
> because they are EOL and no longer tested by the newer driver CI. I don’t 
> think there are actually any changes yet that it won’t work in 3.6 still?  
> Maybe someone with Python 3.6 installed can change the if and see?  I think 
> we have some cqlsh tests in dtest?  As long as we as a project run those on 
> RHEL 7, I would be comfortable with adding that back to being supported. 
> Though maybe just in the rpm package?
> 
> -Jeremiah
> 
>> On Mar 11, 2024, at 1:33 PM, Josh McKenzie  wrote:
>> 
>> 
>> Looks like we bumped from 3.6 requirement to 3.7 in CASSANDRA-18960 
>> <https://issues.apache.org/jira/browse/CASSANDRA-18960> as well - similar 
>> thing. Vector support in python, though that patch took it from "return a 
>> simple blob" to "return something the python driver knows about, but 
>> apparently not variable types so we'll need to upgrade again."
>> 
>>> The version of the Python driver that is used by cqlsh (3.25.0) doesn't 
>>> entirely support the new vector data type introduced by CASSANDRA-18504 
>>> <https://issues.apache.org/jira/browse/CASSANDRA-18504>. While we can 
>>> perfectly write data, read vectors are presented as blobs:
>> 
>> As far as I can tell, support for vector types in cqlsh is the sole reason 
>> we've bumped to 3.7 and 3.8 to support that python driver. That correct 
>> Andres / Brandon?
>> 
>> On Mon, Mar 11, 2024, at 1:22 PM, Caleb Rackliffe wrote:
>>> The vector issues itself was a simple error message change: 
>>> https://github.com/datastax/python-driver/commit/e90c0f5d71f4cac94ed80ed72c8789c0818e11d0
>>> 
>>> Was there something else in 3.29.0 that actually necessitated the move to a 
>>> floor of Python 3.8? Do we generally change runtime requirements in minor 
>>> releases for the driver?
>>> 
>>> On Mon, Mar 11, 2024 at 12:12 PM Brandon Williams >> <mailto:dri...@gmail.com>> wrote:
>>> Given that 3.6 has been EOL for 2+ years[1], I don't think it makes
>>> sense to add support for it back.
>>> 
>>> Kind Regards,
>>> Brandon
>>> 
>>> [1] https://devguide.python.org/versions/
>>> 
>>> On Mon, Mar 11, 2024 at 12:08 PM David Capwell >> <mailto:dcapw...@apple.com>> wrote:
>>> >
>>> > Originally we had planned to support RHEL 7 but in testing 5.0 we found 
>>> > out that cqlsh no longer works on RHEL 7[1].  This was changed in 
>>> > CASSANDRA-19245 which upgraded python-driver from 3.28.0 to 3.29.0. For 
>>> > some reason this minor version upgrade also dropped support for python 
>>> > 3.6 which is the supported python version on RHEL 7.
>>> >
>>> > We wanted to bring this to the attention of the community to figure out 
>>> > next steps; do we wish to say that RHEL 7 is no longer supported (making 
>>> > upgrades tied to OS upgrades, which can be very hard for users), or do we 
>>> > want to add python 3.6 support back to python-driver?
>>> >
>>> >
>>> > 1: the error seen by users is
>>> > $ cqlsh
>>> > Warning: unsupported version of Python, required 3.8-3.11 but found 3.6 
>>> > Warning: unsupported version of Python, required 3.8-3.11 but found 2.7
>>> > No appropriate Python interpreter found.
>>> > $
>>> >
>>> >
>> 

[DISCUSS] Cassandra 5.0 support for RHEL 7

[David Capwell]

Originally we had planned to support RHEL 7 but in testing 5.0 we found out 
that cqlsh no longer works on RHEL 7[1].  This was changed in CASSANDRA-19245 
which upgraded python-driver from 3.28.0 to 3.29.0. For some reason this minor 
version upgrade also dropped support for python 3.6 which is the supported 
python version on RHEL 7.

We wanted to bring this to the attention of the community to figure out next 
steps; do we wish to say that RHEL 7 is no longer supported (making upgrades 
tied to OS upgrades, which can be very hard for users), or do we want to add 
python 3.6 support back to python-driver?


1: the error seen by users is
$ cqlsh
Warning: unsupported version of Python, required 3.8-3.11 but found 3.6 
Warning: unsupported version of Python, required 3.8-3.11 but found 2.7 
No appropriate Python interpreter found.
$ 

[DISCUSS] What SHOULD we do when we index an inet type that is ipv4?

[David Capwell]

So, was reviewing SAI and found we convert ipv4 to ipv6 (which is valid for the 
type) and made me wonder what the behavior would be if client mixed ipv4 with 
ipv4 encoded as ipv6… this caused me to find a different behavior in SAI to the 
rest of C*… where I feel C* is doing the wrong thing…

Lets walk over a simple example

ipv4: 127.0.0.1
ipv6: 0:0:0:0:0::7f00:0001

Both of these address are equal according to networking and java… but for C* 
they are different!  These are 2 different values as ipv4 is 4 bytes and ipv6 
is 16 bytes, so 4 != 16!

With SAI we convert all ipv4 to ipv6 so that the search logic is correct… this 
causes SAI to return partitions that ALLOW FILTERING and other indexes wouldn’t…

This gets to the question in the subject… what SHOULD we do for this type? 

I see 3 options:

1) SAI use the custom C* semantics where 4 != 16… this keeps us consistent… 
2) ALLOW FILTERING and other indexes are “fixed” so that we actually match 
correctly… we are not really able to fix if the type is in a partition or 
clustering column though…
3) deprecate inet in favor of a inet_better type… where inet semantics is the 
custom C* semantics and inet_better handles this case

Thoughts?

Re: Welcome Brad Schoening as Cassandra Committer

[David Capwell]

Congrats!

> On Feb 26, 2024, at 2:14 AM, Mick Semb Wever  wrote:
> 
> Big Congrats Brad! 
> 
> On Wed, 21 Feb 2024 at 21:47, Josh McKenzie  > wrote:
>> The Apache Cassandra PMC is pleased to announce that Brad Schoening has 
>> accepted
>> the invitation to become a committer.
>> 
>> Your work on the integrated python driver, launch script environment, and 
>> tests
>> has been a big help to many. Congratulations and welcome!
>> 
>> The Apache Cassandra PMC members

Re: [Discuss] "Latest" configuration for testing and evaluation (CASSANDRA-18753)

[David Capwell]

I’m +1 once the tests are passing ands +0 while they are failingSent from my iPhoneOn Feb 16, 2024, at 6:08 AM, Paulo Motta  wrote:Thanks for clarifying Branimir! I'm +1 on proceeding as proposed and I think this change will make it easier to gain confidence to update configurations.Interesting discussion and suggestions on this thread - I think we can follow-up on improving test/CI workflow in a different thread/proposal to avoid blocking this.On Thu, Feb 15, 2024 at 9:59 AM Branimir Lambov  wrote:Paulo:1) Will cassandra.yaml remain the default test config? Is the plan moving forward to require green CI for both configurations on pre-commit, or pre-release?The plan is to ensure both configurations are green pre-commit. This should not increase the CI cost as this replaces extra configurations we were running before (e.g. test-tries).2) What will this mean for the release artifact, is the idea to continue shipping with the current cassandra.yaml or eventually switch to the optimized configuration (ie. 6.X) while making the legacy default configuration available via an optional flag?The release simply includes an additional yaml file, which contains a one-liner how to use it.Jeff:1) If there’s an “old compatible default” and “latest recommended settings”, when does the value in “old compatible default” get updated? Never? This does not change anything about these decisions. The question is very serious without this patch as well: Does V6 have to support pain-free upgrade from V5 working in V4 compatible mode? If so, can we ever deprecate or drop anything? If not, are we not breaking upgradeability promises?2) If there are test failures with the new values, it seems REALLY IMPORTANT to make sure those test failures are discovered + fixed IN THE FUTURE TOO. If pushing new yaml into a different file makes us less likely to catch the failures in the future, it seems like we’re hurting ourselves. Branimir mentions this, but how do we ensure that we don’t let this pattern disguise future bugs? The main objective of this patch is to ensure that the second yaml is tested too, pre-commit. We were not doing this for all features we tell users are supported.Paulo:- if cassandra_latest.yaml becomes the new default configuration for 6.0, then precommit only needs to be run against thatversion - prerelease needs to be run against all cassandra.yaml variants.Assuming we keep the pace of development, there will be new "latest" features in 6.0 (e.g. Accord could be one). The idea is more to move some of the settings from latest to default when they are deemed mature enough.Josh:I propose to significantly reduce that stuff. Let's distinguish the packages of tests that need to be run with CDC enabled / disabled, with commitlog compression enabled / disabled, tests that verify sstable formats (mostly io and index I guess), and leave other parameters set as with the latest configuration - this is the easiest way I think. For dtests we have vnodes/no-vnodes, offheap/onheap, and nothing about other stuff. To me running no-vnodes makes no sense because no-vnodes is just a special case of vnodes=1. On the other hand offheap/onheap buffers could be tested in unit tests. In short, I'd run dtests only with the default and latest configuration.Some of these changes are already done in this ticket. Regards,BranimirOn Thu, Feb 15, 2024 at 3:08 PM Paulo Motta  wrote:> It's also been questioned about why we don't just enable settings we recommend.  These are settings we recommend for new clusters.  *Our existing cassandra.yaml needs to be tailored for existing clusters being upgraded, where we are very conservative about changing defaults.*I think this unnecessarily penalizes new users with subpar defaults and existing users who wish to use optimized/recommended defaults and need to maintain additional logic to support that. This change offers an opportunity to revisit this.Is not updating the default cassandra.yaml with new recommended configuration just to protect existing clusters from accidentally overriding cassandra.yaml with a new version during major upgrades? If so, perhaps we could add a new explicit flag “enable_major_upgrade: false” to “cassandra.yaml” that fails startup if an upgrade is detected and force operators to review the configuration before a major upgrade?Related to Jeff’s question, I think we need a way to consolidate “latest recommended settings” into “old compatible default” when cutting a new major version, otherwise the files will diverge perpetually.I think cassandra_latest.yaml offers a way to “buffer” proposals for default configuration changes which are consolidated into “cassandra.yaml” in the subsequent major release, eventually converging configurations and reducing the maintenance burden.On Thu, 15 Feb 2024 at 04:24 Mick Semb Wever  wrote:      Mick and Ekaterina (and everyone really) - any thoughts on what test coverage, if any, we 

Re: [Discuss] "Latest" configuration for testing and evaluation (CASSANDRA-18753)

[David Capwell]

This thread got large quick, yay!

>> is there a reason all guardrails and reliability (aka repair retries) 
>> configs are off by default?  They are off by default in the normal config 
>> for backwards compatibility reasons, but if we are defining a config saying 
>> what we recommend, we should enable these things by default IMO.
> 
> This is one more question to be answered by this discussion. Are there other 
> options that should be enabled by the "latest" configuration? To what values 
> should they be set?
> Is there something that is currently enabled that should not be?

Very likely, we should try to figure that out.  We should also answer how 
conservative do we want to be by default?  There are many configs we need to 
flesh out here, glad to help with the configs I authored (prob best for JIRA 
rather than this thread)

> 
>> Should we merge the configs breaking these tests?  No…. When we have failing 
>> tests people do not spend the time to figure out if their logic caused a 
>> regression and merge, making things more unstable… so when we merge failing 
>> tests that leads to people merging even more failing tests...
> In this case this also means that people will not see at all failures that 
> they introduce in any of the advanced features, as they are not tested at 
> all. Also, since CASSANDRA-19167 and 19168 already have fixes, the non-latest 
> test suite will remain clean after merge. Note that these two problems 
> demonstrate that we have failures in the configuration we ship with, because 
> we are not actually testing it at all. IMHO this is a problem that we should 
> not delay fixing.


I am not arguing we should not get this into CI, but more we should fix the 
known issues before getting into CI… its what we normally do, I don’t see a 
reason to special case this work.

I am 100% cool blocking 5.0 on these bugs found (even if they are test 
failures), but don’t feel we should enable in CI until these issues are 
resolved; we can add the yaml now, but not the CI pipelines.


> 1) If there’s an “old compatible default” and “latest recommended settings”, 
> when does the value in “old compatible default” get updated? Never? 

> How about replacing cassandra.yaml with cassandra_latest.yaml on trunk when 
> cutting cassandra-6.0 branch? Any new default changes on trunk go to 
> cassandra_latest.yaml.

I feel its dangerous to define this at the file level and should do at the 
config level… I personally see us adding new features disabled by default in 
cassandra.yaml and the recommended values in Cassandra-latest.yaml… If I add a 
config in 5.1.2 should it get enabled by default in 6.0?  I don’t feel thats 
wise.

Maybe it makes sense to annotate the configs with the target version for the 
default change?

> Let's distinguish the packages of tests that need to be run with CDC enabled 
> / disabled, with commitlog compression enabled / disabled, tests that verify 
> sstable formats (mostly io and index I guess), and leave other parameters set 
> as with the latest configuration - this is the easiest way I think. 

Yes please!  I really hate having a pipeline per config, we should annotate 
this some how in the tests that matter… junit can param the tests for us so we 
cover the different configs the test supports… I have written many tests that 
are costly and run on all these other pipelines but have 0 change in the 
config… just wasting resources rerunning…

Pushing this to the test also is a better author/maintainer experience… running 
the test in your IDE and seeing all the params and their results is so much 
better than monkeying around with yaml files and ant…. My repair simulation 
tests have a hack flag to try to switch the yaml to make it easier to test 
against the other configs and I loath it so much…

> To me running no-vnodes makes no sense because no-vnodes is just a special 
> case of vnodes=1

And to many in the community the only config for production =).  In this debate 
we have 3 camps: no-vnode, vnode <= 4 tokens, vnode > 4 tokens (I am 
simplifying….)… For those in the no-vnode camp their tests are focused on this 
case and get disabled when vnodes are enabled (so not running this config 
lowers coverage).  

I don’t think we are going to solve this debate in this thread, but if we can 
push this to the test to run as a param I think thats best… avoid having 2 
pipelines and push this to the tests that actually support both configs...

> On Feb 15, 2024, at 10:20 AM, Jon Haddad  wrote:
> 
> For the sake of argument, if we picked one, would you (anyone, not just 
> Stefan) be OK with the JVM that's selected being the one you don't use?  I'm 
> willing to bet most folks will have a preference for the JVM they run in 
> production.  I think both should be run on some frequent basis but I can 
> definitely see the reasoning behind not wanting it to block folks on work, it 
> sounds like a lot of wasted days waiting on CI especially during a bigger 
> 

Re: [Discuss] "Latest" configuration for testing and evaluation (CASSANDRA-18753)

[David Capwell]

> so can cause repairs to deadlock forever

Small correction, I finished fixing the tests in CASSANDRA-19042 and we don’t 
deadlock, we timeout and fail repair if any of those messages are dropped.  

> On Feb 13, 2024, at 11:04 AM, David Capwell  wrote:
> 
>> and to point potential users that are evaluating the technology to an 
>> optimized set of defaults
> 
> Left this comment in the GH… is there a reason all guardrails and reliability 
> (aka repair retries) configs are off by default?  They are off by default in 
> the normal config for backwards compatibility reasons, but if we are defining 
> a config saying what we recommend, we should enable these things by default 
> IMO.
> 
>> There are currently a number of test failures when the new options are 
>> selected, some of which appear to be genuine problems. Is the community okay 
>> with committing the patch before all of these are addressed?
> 
> I was tagged on CASSANDRA-19042, the paxos repair message handing does not 
> have the repair reliably improvements that 5.0 have, so can cause repairs to 
> deadlock forever (same as current 4.x repairs).  Bringing these up to par 
> with the rest of repair would be very much welcome (they are also lacking 
> visibility, so need to fallback to heap dumps to see what’s going on; same as 
> 4.0.x but not 4.1.x), but I doubt I have cycles to do that…. This refactor is 
> not 100% trivial as it has fun subtle concurrency issues to address (message 
> retries and dedupping), and making sure this logic works with the existing 
> repair simulation tests does require refactoring how the paxos cleanup state 
> is tracked, which could have subtle consequents.
> 
> I do think this should be fixed, but should it block 5.0?  Not sure… will 
> leave to others….
> 
> Should we merge the configs breaking these tests?  No…. When we have failing 
> tests people do not spend the time to figure out if their logic caused a 
> regression and merge, making things more unstable… so when we merge failing 
> tests that leads to people merging even more failing tests...
> 
>> On Feb 13, 2024, at 8:41 AM, Branimir Lambov  wrote:
>> 
>> Hi All,
>> 
>> CASSANDRA-18753 introduces a second set of defaults (in a separate 
>> "cassandra_latest.yaml") that enable new features of Cassandra. The 
>> objective is two-fold: to be able to test the database in this 
>> configuration, and to point potential users that are evaluating the 
>> technology to an optimized set of defaults that give a clearer picture of 
>> the expected performance of the database for a new user. The objective is to 
>> get this configuration into 5.0 to have the extra bit of confidence that we 
>> are not releasing (and recommending) options that have not gone through 
>> thorough CI.
>> 
>> The implementation has already gone through review, but I'd like to get 
>> people's opinion on two things:
>> - There are currently a number of test failures when the new options are 
>> selected, some of which appear to be genuine problems. Is the community okay 
>> with committing the patch before all of these are addressed? This should 
>> prevent the introduction of new failures and make sure we don't release 
>> before clearing the existing ones.
>> - I'd like to get an opinion on what's suitable wording and documentation 
>> for the new defaults set. Currently, the patch proposes adding the following 
>> text to the yaml (see https://github.com/apache/cassandra/pull/2896/files):
>> # NOTE:
>> #   This file is provided in two versions:
>> # - cassandra.yaml: Contains configuration defaults for a "compatible"
>> #   configuration that operates using settings that are 
>> backwards-compatible
>> #   and interoperable with machines running older versions of Cassandra.
>> #   This version is provided to facilitate pain-free upgrades for 
>> existing
>> #   users of Cassandra running in production who want to gradually and
>> #   carefully introduce new features.
>> # - cassandra_latest.yaml: Contains configuration defaults that enable
>> #   the latest features of Cassandra, including improved functionality as
>> #   well as higher performance. This version is provided for new users of
>> #   Cassandra who want to get the most out of their cluster, and for 
>> users
>> #   evaluating the technology.
>> #   To use this version, simply copy this file over cassandra.yaml, or 
>> specify
>> #   it using the -Dcassandra.config system property, e.g. by running
>> # cassandra 
>> -Dcassandra.config=file:/$CASSANDRA_HOME/conf/cassandra_latest.yaml
>> # /NOTE
>> Does this sound sensible? Should we add a pointer to this defaults set 
>> elsewhere in the documentation?
>> 
>> Regards,
>> Branimir
> 

Re: [Discuss] "Latest" configuration for testing and evaluation (CASSANDRA-18753)

[David Capwell]

> and to point potential users that are evaluating the technology to an 
> optimized set of defaults

Left this comment in the GH… is there a reason all guardrails and reliability 
(aka repair retries) configs are off by default?  They are off by default in 
the normal config for backwards compatibility reasons, but if we are defining a 
config saying what we recommend, we should enable these things by default IMO.

> There are currently a number of test failures when the new options are 
> selected, some of which appear to be genuine problems. Is the community okay 
> with committing the patch before all of these are addressed?

I was tagged on CASSANDRA-19042, the paxos repair message handing does not have 
the repair reliably improvements that 5.0 have, so can cause repairs to 
deadlock forever (same as current 4.x repairs).  Bringing these up to par with 
the rest of repair would be very much welcome (they are also lacking 
visibility, so need to fallback to heap dumps to see what’s going on; same as 
4.0.x but not 4.1.x), but I doubt I have cycles to do that…. This refactor is 
not 100% trivial as it has fun subtle concurrency issues to address (message 
retries and dedupping), and making sure this logic works with the existing 
repair simulation tests does require refactoring how the paxos cleanup state is 
tracked, which could have subtle consequents.

I do think this should be fixed, but should it block 5.0?  Not sure… will leave 
to others….

Should we merge the configs breaking these tests?  No…. When we have failing 
tests people do not spend the time to figure out if their logic caused a 
regression and merge, making things more unstable… so when we merge failing 
tests that leads to people merging even more failing tests...

> On Feb 13, 2024, at 8:41 AM, Branimir Lambov  wrote:
> 
> Hi All,
> 
> CASSANDRA-18753 introduces a second set of defaults (in a separate 
> "cassandra_latest.yaml") that enable new features of Cassandra. The objective 
> is two-fold: to be able to test the database in this configuration, and to 
> point potential users that are evaluating the technology to an optimized set 
> of defaults that give a clearer picture of the expected performance of the 
> database for a new user. The objective is to get this configuration into 5.0 
> to have the extra bit of confidence that we are not releasing (and 
> recommending) options that have not gone through thorough CI.
> 
> The implementation has already gone through review, but I'd like to get 
> people's opinion on two things:
> - There are currently a number of test failures when the new options are 
> selected, some of which appear to be genuine problems. Is the community okay 
> with committing the patch before all of these are addressed? This should 
> prevent the introduction of new failures and make sure we don't release 
> before clearing the existing ones.
> - I'd like to get an opinion on what's suitable wording and documentation for 
> the new defaults set. Currently, the patch proposes adding the following text 
> to the yaml (see https://github.com/apache/cassandra/pull/2896/files):
> # NOTE:
> #   This file is provided in two versions:
> # - cassandra.yaml: Contains configuration defaults for a "compatible"
> #   configuration that operates using settings that are 
> backwards-compatible
> #   and interoperable with machines running older versions of Cassandra.
> #   This version is provided to facilitate pain-free upgrades for existing
> #   users of Cassandra running in production who want to gradually and
> #   carefully introduce new features.
> # - cassandra_latest.yaml: Contains configuration defaults that enable
> #   the latest features of Cassandra, including improved functionality as
> #   well as higher performance. This version is provided for new users of
> #   Cassandra who want to get the most out of their cluster, and for users
> #   evaluating the technology.
> #   To use this version, simply copy this file over cassandra.yaml, or 
> specify
> #   it using the -Dcassandra.config system property, e.g. by running
> # cassandra 
> -Dcassandra.config=file:/$CASSANDRA_HOME/conf/cassandra_latest.yaml
> # /NOTE
> Does this sound sensible? Should we add a pointer to this defaults set 
> elsewhere in the documentation?
> 
> Regards,
> Branimir

Re: Welcome Maxim Muzafarov as Cassandra Committer

[David Capwell]

Congrats!

> On Jan 8, 2024, at 10:53 AM, Jacek Lewandowski  
> wrote:
> 
> Congratulations Maxim, well deserved, it's a pleasure to work with you!
> 
> - - -- --- -  -
> Jacek Lewandowski
> 
> 
> pon., 8 sty 2024 o 19:35 Lorina Poland  > napisał(a):
>> Congratulations Maxim!
>> 
>> On 2024/01/08 18:19:04 Josh McKenzie wrote:
>> > The Apache Cassandra PMC is pleased to announce that Maxim Muzafarov has 
>> > accepted
>> > the invitation to become a committer.
>> > 
>> > Thanks for all the hard work and collaboration on the project thus far, 
>> > and we're all looking forward to working more with you in the future. 
>> > Congratulations and welcome!
>> > 
>> > The Apache Cassandra PMC members
>> > 
>> > 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[David Capwell]

 all the statistics of its memtable and SSTables. For the 
>>>> distribution of these node statistics we are still exploring different 
>>>> options. We can come back with a precise proposal once we have hammered 
>>>> all the details. 
>>>> Is it for you a blocker for this CEP or do you just want to make sure that 
>>>> this part is discussed in deeper details before we implement it?
>>>> 
>>>> 
>>>>  
>>>> The proposal also seems to imply we are aiming for coordinators to all 
>>>> make the same decision for a query, which I think is challenging, and it 
>>>> would be worth fleshing out the design here a little (perhaps just in 
>>>> Jira).
>>>> 
>>>> 
>>>> The goal is that the large majority of nodes preparing a query at a given 
>>>> point in time should make the same decision and that over time all nodes 
>>>> should converge toward the same decision. This part is dependent on the 
>>>> node statistics distribution, the cost model and the triggers for 
>>>> re-optimization (that will require some experimentation).
>>>> 
>>>> There’s also not much discussion of the execution model: I think it would 
>>>> make most sense for this to be independent of any cost and optimiser 
>>>> models (though they might want to operate on them), so that EXPLAIN and 
>>>> hints can work across optimisers (a suitable hint might essentially bypass 
>>>> the optimiser, if the optimiser permits it, by providing a standard 
>>>> execution model)
>>>> 
>>>> It is not clear to me what you mean by "a standard execution model"? 
>>>> Otherwise, we were not planning to have the execution model or the hints 
>>>> depending on the optimizer. 
>>>> 
>>>> I think it would be worth considering providing the execution plan to the 
>>>> client as part of query preparation, as an opaque payload to supply to 
>>>> coordinators on first contact, as this might simplify the problem of 
>>>> ensuring queries behave the same without adopting a lot of complexity for 
>>>> synchronising statistics (which will never provide strong guarantees). Of 
>>>> course, re-preparing a query might lead to a new plan, though any 
>>>> coordinators with the query in their cache should be able to retrieve it 
>>>> cheaply. If the execution model is efficiently serialised this might have 
>>>> the ancillary benefit of improving the occupancy of our prepared query 
>>>> cache.
>>>> 
>>>> I am not sure that I understand your proposal. If 2 nodes build a 
>>>> different execution plan how do you solve that conflict?
>>>> 
>>>> Le mer. 13 déc. 2023 à 09:55, Benedict >>> <mailto:bened...@apache.org>> a écrit :
>>>> 
>>>> A CBO can only make worse decisions than the status quo for what I presume 
>>>> are the majority of queries - i.e. those that touch only primary indexes. 
>>>> In general, there are plenty of use cases that prefer determinism. So I 
>>>> agree that there should at least be a CBO implementation that makes the 
>>>> same decisions as the status quo, deterministically.
>>>> 
>>>> I do support the proposal, but would like to see some elements discussed 
>>>> in more detail. The maintenance and distribution of summary statistics in 
>>>> particular is worthy of its own CEP, and it might be preferable to split 
>>>> it out. The proposal also seems to imply we are aiming for coordinators to 
>>>> all make the same decision for a query, which I think is challenging, and 
>>>> it would be worth fleshing out the design here a little (perhaps just in 
>>>> Jira).
>>>> 
>>>> While I’m not a fan of ALLOW FILTERING, I’m not convinced that this CEP 
>>>> deprecates it. It is a concrete qualitative guard rail, that I expect some 
>>>> users will prefer to a cost-based guard rail. Perhaps this could be left 
>>>> to the CBO to decide how to treat.
>>>> 
>>>> There’s also not much discussion of the execution model: I think it would 
>>>> make most sense for this to be independent of any cost and optimiser 
>>>> models (though they might want to operate on them), so that EXPLAIN and 
>>>> hints can work across optimisers (a suitable hint might essentially bypass 
>>>> the optimiser, if the optimiser permits it, by providing a standard 
>>>> execution model)
>>>> 
>>>> I think it would be worth considering providing the execution plan to the 
>>>> client as part of query preparation, as an opaque payload to supply to 
>>>> coordinators on first contact, as this might simplify the problem of 
>>>> ensuring queries behave the same without adopting a lot of complexity for 
>>>> synchronising statistics (which will never provide strong guarantees). Of 
>>>> course, re-preparing a query might lead to a new plan, though any 
>>>> coordinators with the query in their cache should be able to retrieve it 
>>>> cheaply. If the execution model is efficiently serialised this might have 
>>>> the ancillary benefit of improving the occupancy of our prepared query 
>>>> cache.
>>>> 
>>>> 
>>>>> On 13 Dec 2023, at 00:44, Jon Haddad >>>> <mailto:j...@jonhaddad.com>> wrote:
>>>>> 
>>>>> I think it makes sense to see what the actual overhead is of CBO before 
>>>>> making the assumption it'll be so high that we need to have two code 
>>>>> paths.  I'm happy to provide thorough benchmarking and analysis when it 
>>>>> reaches a testing phase.  
>>>>> 
>>>>> I'm excited to see where this goes.  I think it sounds very forward 
>>>>> looking and opens up a lot of possibilities.
>>>>> 
>>>>> Jon
>>>>> 
>>>>> On Tue, Dec 12, 2023 at 4:25 PM guo Maxwell >>>> <mailto:cclive1...@gmail.com>> wrote:
>>>>> Nothing expresses my thoughts better than +1
>>>>> ，It feels like it means a lot to Cassandra.
>>>>> 
>>>>> I have a question. Is it easy to turn off cbo's optimizer or by pass in 
>>>>> some way? Because some simple read and write requests will have better 
>>>>> performance without cbo, which is also the advantage of Cassandra 
>>>>> compared to some rdbms.
>>>>> 
>>>>> David Capwell >>>> <mailto:dcapw...@apple.com>>于2023年12月13日 周三上午3:37写道：
>>>>> Overall LGTM.  
>>>>> 
>>>>> 
>>>>>> On Dec 12, 2023, at 5:29 AM, Benjamin Lerer >>>>> <mailto:ble...@apache.org>> wrote:
>>>>>> 
>>>>>> Hi everybody,
>>>>>> 
>>>>>> I would like to open the discussion on the introduction of a cost based 
>>>>>> optimizer to allow Cassandra to pick the best execution plan based on 
>>>>>> the data distribution.Therefore, improving the overall query performance.
>>>>>> 
>>>>>> This CEP should also lay the groundwork for the future addition of 
>>>>>> features like joins, subqueries, OR/NOT and index ordering.
>>>>>> 
>>>>>> The proposal is here: 
>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-39%3A+Cost+Based+Optimizer
>>>>>> 
>>>>>> Thank you in advance for your feedback.

Re: Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?

[David Capwell]

> A brief perusal shows jqwik as integrated with JUnit 5 taking a fairly 
> interesting annotation-based approach to property testing. Curious if you've 
> looked into or used that at all David (Capwell)? (link for the lazy: 
> https://jqwik.net/docs/current/user-guide.html#detailed-table-of-contents).

I have not no.  Looking at your link it moves from lambdas to annotations, and 
tries to define a API for stateful… I am neutral to that as its mostly style…. 
One thing to call out is that the project documents it tries to “shrink”… we 
ended up disabling this in QuickTheories as shrinking doesn’t work well for 
many of our tests (too high resource demand and unable to actually shrink once 
you move past trivial generators).  Looking at their docs and their code, its 
hard for me to see how we actually create C* generators… its so much class gen 
magic that I really don’t see how to create AbstractType or TableMetadata… the 
only example they gave was not random data but hand crafted data… 

> moving to JUnit 5

I am a fan of this.  If we add dependencies and don’t keep update with them it 
becomes painful over time (missing features, lack of support, etc).  

> First of all - when you want to have a parameterized test case you do not 
> have to make the whole test class parameterized - it is per test case. Also, 
> each method can have different parameters.

I strongly prefer this, but never had it as a blocker from me doing param 
tests…. One thing where this “could” come into play is that we currently run 
with different configs at the CI level and we might be able to make this happen 
at the class or method level instead..

@ServerConfigs(all) // can exclude unsupported configs
public class InsertTest

It bothers me deeply that we run tests that don’t touch the configs we use in 
CI, causing us to waste resources… Can we solve this in junit4 param logic… no 
clue… 

> On Dec 15, 2023, at 6:52 PM, Josh McKenzie  wrote:
> 
>> First of all - when you want to have a parameterized test case you do not 
>> have to make the whole test class parameterized - it is per test case. Also, 
>> each method can have different parameters.
> This is a pretty compelling improvement to me having just had to use the 
> somewhat painful and blunt instrument of our current framework's 
> parameterization; pretty clunky and broad.
> 
> It also looks like they moved to a "test engine abstracted away from test 
> identification" approach to their architecture in 5 w/the "vintage" model 
> providing native unchanged backwards-compatibility w/junit 4. Assuming they 
> didn't bork up their architecture that should lower risk of the framework 
> change leading to disruption or failure (famous last words...).
> 
> A brief perusal shows jqwik as integrated with JUnit 5 taking a fairly 
> interesting annotation-based approach to property testing. Curious if you've 
> looked into or used that at all David (Capwell)? (link for the lazy: 
> https://jqwik.net/docs/current/user-guide.html#detailed-table-of-contents).
> 
> On Tue, Dec 12, 2023, at 11:39 AM, Jacek Lewandowski wrote:
>> First of all - when you want to have a parameterized test case you do not 
>> have to make the whole test class parameterized - it is per test case. Also, 
>> each method can have different parameters.
>> 
>> For the extensions - we can have extensions which provide Cassandra 
>> configuration, extensions which provide a running cluster and others. We 
>> could for example apply some extensions to all test classes externally 
>> without touching those classes, something like logging the begin and end of 
>> each test case. 
>> 
>> 
>> 
>> wt., 12 gru 2023 o 12:07 Benedict > <mailto:bened...@apache.org>> napisał(a):
>> 
>> Could you give (or link to) some examples of how this would actually benefit 
>> our test suites?
>> 
>> 
>>> On 12 Dec 2023, at 10:51, Jacek Lewandowski >> <mailto:lewandowski.ja...@gmail.com>> wrote:
>>> 
>>> I have two major pros for JUnit 5:
>>> - much better support for parameterized tests
>>> - global test hooks (automatically detectable extensions) + 
>>> multi-inheritance
>>> 
>>> 
>>> 
>>> 
>>> pon., 11 gru 2023 o 13:38 Benedict >> <mailto:bened...@apache.org>> napisał(a):
>>> 
>>> Why do we want to move to JUnit 5? 
>>> 
>>> I’m generally opposed to churn unless well justified, which it may be - 
>>> just not immediately obvious to me.
>>> 
>>> 
>>>> On 11 Dec 2023, at 08:33, Jacek Lewandowski >>> <mailto:lewandowski.ja...@gmail.com>> wrote:
>>>> 
>>>> N

Re: Moving Semver4j from test to main dependencies

[David Capwell]

+1

> On Dec 15, 2023, at 7:35 PM, Mick Semb Wever  wrote:
> 
> 
> 
>> I'd like to add Semver4j to the production dependencies. It is currently on 
>> the test classpath. The library is pretty lightweight, licensed with MIT and 
>> has no transitive dependencies.
>> 
>> We need to represent the kernel version somehow in CASSANDRA-19196 and 
>> Semver4j looks as the right tool for it. Maybe at some point we can replace 
>> our custom implementation of CassandraVersion as well. 
> 
> 
> I'm +1 on both counts.
> 
> But IMO you need to include those that were involved in this past discussion 
> that touched on the use of this library:  
> https://lists.apache.org/thread/zz3x1zl1lo8rkqpf0cl992y6fsy4r9gc 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[David Capwell]

Overall LGTM.  

> On Dec 12, 2023, at 5:29 AM, Benjamin Lerer  wrote:
> 
> Hi everybody,
> 
> I would like to open the discussion on the introduction of a cost based 
> optimizer to allow Cassandra to pick the best execution plan based on the 
> data distribution.Therefore, improving the overall query performance.
> 
> This CEP should also lay the groundwork for the future addition of features 
> like joins, subqueries, OR/NOT and index ordering.
> 
> The proposal is here: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-39%3A+Cost+Based+Optimizer
> 
> Thank you in advance for your feedback.

Re: Welcome Mike Adamson as Cassandra committer

[David Capwell]

Congrats!

> On Dec 8, 2023, at 11:00 AM, Lorina Poland  wrote:
> 
> Congratulations, Mike!

Re: [DISCUSS] CASSANDRA-19113: Publishing dtest-shaded JARs on release

[David Capwell]

+1 from me

> On Nov 28, 2023, at 12:55 PM, Doug Rohrer  wrote:
> 
> +1 (nb, but not a vote, so ¯\_(ツ)_/¯ ) - would be lovely to not have to deal 
> with this individually for each project in which we use the in-jvm dtest 
> framework. As Francisco noted, we’re using this in the sidecar and Analytics 
> projects now and I’ve had to jump through a lot of hoops to get everything 
> building consistently.
> 
> I’ve got some minor modifications to the way in which the existing shading 
> works that I can contribute back to the core Cassandra project (mostly, a few 
> additional relocations and not using the user’s default Maven cache as the 
> temporary installation location as it was difficult to make sure you had the 
> correct dtest jar with a bunch of them in the `.m2` directory).
> 
> Doug
> 
>> On Nov 28, 2023, at 2:51 PM, Josh McKenzie  wrote:
>> 
>> Building these jars every time we run every CI job is just silly.
>> 
>> +1.
>> 
>> On Tue, Nov 28, 2023, at 2:08 PM, Francisco Guerrero wrote:
>>> Hi Abe,
>>> 
>>> I'm +1 on this. Several Cassandra-ecosystem projects build the dtest jar in 
>>> CI. We'd very
>>> much prefer to just consumed shaded dtest jars from Cassandra releases for 
>>> testing
>>> purposes.
>>> 
>>> Best,
>>> - Francisco
>>> 
>>> On 2023/11/28 19:02:17 Abe Ratnofsky wrote:
>>> > Hey folks - wanted to raise a separate thread to discuss publishing of 
>>> > dtest-shaded JARs on release.
>>> > 
>>> > Currently, adjacent projects that want to use the jvm-dtest framework 
>>> > need to build the shaded JARs themselves. This is a decent amount of 
>>> > work, and is duplicated across each project. This is mainly relevant for 
>>> > projects like Sidecar and Driver. Currently, those projects need to clone 
>>> > and build apache/cassandra themselves, run ant dtest-jar, and move the 
>>> > JAR into the appropriate place. Different build systems treat local JARs 
>>> > differently, and the whole process can be a bit complicated. Would be 
>>> > great to be able to treat these as normal dependencies.
>>> > 
>>> > https://issues.apache.org/jira/browse/CASSANDRA-19113
>>> > 
>>> > Any objections?
>>> > 
>>> > --
>>> > Abe
> 

Re: [DISCUSS] Harry in-tree

[David Capwell]

+1 to in-tree

> On Nov 27, 2023, at 9:17 AM, Benjamin Lerer  wrote:
> 
> +1
> 
> Le lun. 27 nov. 2023 à 18:01, Brandon Williams  > a écrit :
>> I am +1 on including Harry in-tree.
>> 
>> Kind Regards,
>> Brandon
>> 
>> On Fri, Nov 24, 2023 at 9:44 AM Alex Petrov > > wrote:
>> >
>> > Hi everyone,
>> >
>> > With TCM landed, there will be way more Harry tests in-tree: we are using 
>> > it for many coordination tests, and there's now a simulator test that uses 
>> > Harry. During development, Harry has allowed us to uncover and resolve 
>> > numerous elusive edge cases.
>> >
>> > I had conversations with several folks, and wanted to propose to move 
>> > harry-core to Cassandra test tree. This will substantially 
>> > simplify/streamline co-development of Cassandra and Harry. With a new 
>> > HistoryBuilder API that has helped to find and trigger [1] [2] and [3], it 
>> > will also be much more approachable.
>> >
>> > Besides making it easier for everyone to develop new fuzz tests, it will 
>> > also substantially lower the barrier to entry. Currently, debugging an 
>> > issue found by Harry involves a cumbersome process of rebuilding and 
>> > transferring jars between Cassandra and Harry, depending on which side you 
>> > modify. This not only hampers efficiency but also deters broader adoption. 
>> > By merging harry-core into the Cassandra test tree, we eliminate this 
>> > barrier.
>> >
>> > Thank you,
>> > --Alex
>> >
>> > [1] https://issues.apache.org/jira/browse/CASSANDRA-19011
>> > [2] https://issues.apache.org/jira/browse/CASSANDRA-18993
>> > [3] https://issues.apache.org/jira/browse/CASSANDRA-18932

Re: [DISCUSS] Backport CASSANDRA-18816 to 5.0? Add support for repair coordinator to retry messages that timeout

[David Capwell]

IR patch is up for review https://issues.apache.org/jira/browse/CASSANDRA-18962

> On Oct 24, 2023, at 3:15 PM, David Capwell  wrote:
> 
> I sat down to add IR messages to the mix… given how positive the feedback was 
> for other repair messages I assume people are still ok with this new IR work 
> going to 5.0 as well, if not please let me know here (will send a patch 
> tomorrow).
> 
> Once I send out the patch 100% of repair messages have retry logic
> 
>> On Sep 26, 2023, at 12:08 PM, David Capwell  wrote:
>> 
>> Thanks all for the feedback!  The patch has 2 +1s on trunk and back ported 
>> to 5.0, making sure it’s stable now; I plan to merge early this week.
>> 
>>> On Sep 21, 2023, at 2:07 PM, Ekaterina Dimitrova  
>>> wrote:
>>> 
>>> +1 from me too. Moreover, this work has started as part of the test efforts 
>>> and identifying weak points during the 4.0 testing, if I recall correctly. 
>>> 5.0 sounds like a good place to land. Thank you David and everyone else 
>>> involved for your efforts!
>>> 
>>> On Thu, 21 Sep 2023 at 1:01, Berenguer Blasi >> <mailto:berenguerbl...@gmail.com>> wrote:
>>>> +1 I agree with Brandon. It's more like a bug imo.
>>>> On 20/9/23 21:42, Caleb Rackliffe wrote:
>>>>> +1 on a 5.0 backport
>>>>> 
>>>>> On Wed, Sep 20, 2023 at 2:26 PM Brandon Williams >>>> <mailto:dri...@gmail.com>> wrote:
>>>>>> I think it could be argued that not retrying messages is a bug, I am
>>>>>> +1 on including this in 5.0.
>>>>>> 
>>>>>> Kind Regards,
>>>>>> Brandon
>>>>>> 
>>>>>> On Tue, Sep 19, 2023 at 1:16 PM David Capwell >>>>> <mailto:dcapw...@apple.com>> wrote:
>>>>>> >
>>>>>> > To try to get repair more stable, I added optional retry logic (patch 
>>>>>> > is still in review) to a handful of critical repair verbs.  This patch 
>>>>>> > is disabled by default but allows you to opt-in to retries so 
>>>>>> > ephemeral issues don’t cause a repair to fail after running for a long 
>>>>>> > time (assuming they resolve within the retry window). There are 2 
>>>>>> > protocol level changes to enable this: VALIDATION_RSP and SYNC_RSP now 
>>>>>> > send an ACK (if the sender doesn’t attach a callback, these ACKs get 
>>>>>> > ignored in all versions; see org.apache.cassandra.net 
>>>>>> > <http://org.apache.cassandra.net/>.ResponseVerbHandler#doVerb and 
>>>>>> > Verb.REPAIR_RSP).  Given that we have already forked, I believe we 
>>>>>> > would need to give a waiver to allow this patch due to this change.
>>>>>> >
>>>>>> > The patch was written on trunk, but figured back porting 5.0 would be 
>>>>>> > rather trivial and this was brought up during the review, so floating 
>>>>>> > this to a wider audience.
>>>>>> >
>>>>>> > If you look at the patch you will see that it is very large, but this 
>>>>>> > is only to make testing of repair coordination easier and 
>>>>>> > deterministic, the biggest code changes are:
>>>>>> >
>>>>>> > 1) Moving from ActiveRepairService.instance to 
>>>>>> > ActiveRepairService.instance() (this is the main reason so many files 
>>>>>> > were touched; this was needed so unit tests don’t load the whole world)
>>>>>> > 2) Repair no longer reaches into global space and instead is provided 
>>>>>> > the subsystems needed to perform repair; this change is local to 
>>>>>> > repair code
>>>>>> >
>>>>>> > Both of these changes were only for testing as they allow us to 
>>>>>> > simulate 1k repairs in around 15 seconds with 100% deterministic 
>>>>>> > execution.
>> 
> 

Re: [DISCUSS] Backport CASSANDRA-18816 to 5.0? Add support for repair coordinator to retry messages that timeout

[David Capwell]

I sat down to add IR messages to the mix… given how positive the feedback was 
for other repair messages I assume people are still ok with this new IR work 
going to 5.0 as well, if not please let me know here (will send a patch 
tomorrow).

Once I send out the patch 100% of repair messages have retry logic

> On Sep 26, 2023, at 12:08 PM, David Capwell  wrote:
> 
> Thanks all for the feedback!  The patch has 2 +1s on trunk and back ported to 
> 5.0, making sure it’s stable now; I plan to merge early this week.
> 
>> On Sep 21, 2023, at 2:07 PM, Ekaterina Dimitrova  
>> wrote:
>> 
>> +1 from me too. Moreover, this work has started as part of the test efforts 
>> and identifying weak points during the 4.0 testing, if I recall correctly. 
>> 5.0 sounds like a good place to land. Thank you David and everyone else 
>> involved for your efforts!
>> 
>> On Thu, 21 Sep 2023 at 1:01, Berenguer Blasi > <mailto:berenguerbl...@gmail.com>> wrote:
>>> +1 I agree with Brandon. It's more like a bug imo.
>>> On 20/9/23 21:42, Caleb Rackliffe wrote:
>>>> +1 on a 5.0 backport
>>>> 
>>>> On Wed, Sep 20, 2023 at 2:26 PM Brandon Williams >>> <mailto:dri...@gmail.com>> wrote:
>>>>> I think it could be argued that not retrying messages is a bug, I am
>>>>> +1 on including this in 5.0.
>>>>> 
>>>>> Kind Regards,
>>>>> Brandon
>>>>> 
>>>>> On Tue, Sep 19, 2023 at 1:16 PM David Capwell >>>> <mailto:dcapw...@apple.com>> wrote:
>>>>> >
>>>>> > To try to get repair more stable, I added optional retry logic (patch 
>>>>> > is still in review) to a handful of critical repair verbs.  This patch 
>>>>> > is disabled by default but allows you to opt-in to retries so ephemeral 
>>>>> > issues don’t cause a repair to fail after running for a long time 
>>>>> > (assuming they resolve within the retry window). There are 2 protocol 
>>>>> > level changes to enable this: VALIDATION_RSP and SYNC_RSP now send an 
>>>>> > ACK (if the sender doesn’t attach a callback, these ACKs get ignored in 
>>>>> > all versions; see org.apache.cassandra.net 
>>>>> > <http://org.apache.cassandra.net/>.ResponseVerbHandler#doVerb and 
>>>>> > Verb.REPAIR_RSP).  Given that we have already forked, I believe we 
>>>>> > would need to give a waiver to allow this patch due to this change.
>>>>> >
>>>>> > The patch was written on trunk, but figured back porting 5.0 would be 
>>>>> > rather trivial and this was brought up during the review, so floating 
>>>>> > this to a wider audience.
>>>>> >
>>>>> > If you look at the patch you will see that it is very large, but this 
>>>>> > is only to make testing of repair coordination easier and 
>>>>> > deterministic, the biggest code changes are:
>>>>> >
>>>>> > 1) Moving from ActiveRepairService.instance to 
>>>>> > ActiveRepairService.instance() (this is the main reason so many files 
>>>>> > were touched; this was needed so unit tests don’t load the whole world)
>>>>> > 2) Repair no longer reaches into global space and instead is provided 
>>>>> > the subsystems needed to perform repair; this change is local to repair 
>>>>> > code
>>>>> >
>>>>> > Both of these changes were only for testing as they allow us to 
>>>>> > simulate 1k repairs in around 15 seconds with 100% deterministic 
>>>>> > execution.
> 

Re: CASSANDRA-18775 (Cassandra supported OSs)

[David Capwell]

+1 to drop the whole lib… 

> On Oct 20, 2023, at 7:55 AM, Jeremiah Jordan  
> wrote:
> 
> Agreed.  -1 on selectively removing any of the libs.  But +1 for removing the 
> whole thing if it is no longer used.
> 
> -Jeremiah
> 
> On Oct 20, 2023 at 9:28:55 AM, Mick Semb Wever  > wrote:
>>> Does anyone see any reason _not_ to do this?
>> 
>> 
>> Thanks for bring this to dev@
>> 
>> I see reason not to do it, folk do submit patches for other archs despite us 
>> not formally maintaining and testing the code for those archs.  Some 
>> examples are PPC64 Big Endian (CASSANDRA-7476), s390x (CASSANDRA-17723), 
>> PPC64 Little Endian (CASSANDRA-7381), sparcv9 (CASSANDRA-6628).  Wrote this 
>> on the ticket too.
>> 
>> +1 for removing sigar altogether (as Brandon points out). 
>> 

Re: [DISCUSS] Gossip shutdown may corrupt peers making it so the cluster never converges, and a small protocol change to fix

[David Capwell]

Brandon and I have been talking in CASSANDRA-18913 and here is the current 
plan; sharing for visibility

There are two bugs: 
1) restart and seeing a shutdown event before gossip has settled for you will 
create a partial EndpointState which leads to failed startup
2) shutdown corrupts state due to every node modifying their local copy (this 
may impact host replacements)

For #1 this can be fixed without changing networking, so will create a small 
patch for 4.x line
For #2 it does require a network change, so will add this to 5.0 with mixed 
mode support 

> On Oct 6, 2023, at 4:18 PM, David Capwell  wrote:
> 
>> Won't the replacement have a newer generation?
> 
> The replacement is a different instance.  I performs a shadow round with its 
> seeds and if they are impacted by this issue then they are missing tokens, so 
> we fail the host replacement… you can work around this by changing the seeds 
> to nodes that know the token.
> 
>> I don't think it is, this is just fixing a gossip bug, and we should do so 
>> in all affected versions.
> 
> Right now we sent NoPayload which is 0 bytes, but with the change we send 
> GossipShutdown which contains the whole EndpointState… I “feel” like 4.x can 
> not handle this but worth a test (after deserializing the message we have 
> extra bytes… won’t we get mad about that?)… the gossip handler doesn’t look 
> at the payload so as long as 4.x serialization can support this, then it 
> won’t be hard to back port to 4.x
> 
> 
>> On Oct 6, 2023, at 3:57 PM, Brandon Williams  wrote:
>> 
>> On Fri, Oct 6, 2023 at 5:50 PM David Capwell  wrote:
>>> Lets say you now need to host replace node1
>> 
>> Won't the replacement have a newer generation?
>> 
>>> avoid peers mutating endpoint states they don’t own
>> 
>> This sounds reasonable to me.
>> 
>>> This would be a protocol change, so would need to make sure everyone is 
>>> cool with me doing this in 5.0.>
>> 
>> I don't think it is, this is just fixing a gossip bug, and we should
>> do so in all affected versions.
>> 
>> Kind Regards,
>> Brandon
> 

Re: [DISCUSS] Gossip shutdown may corrupt peers making it so the cluster never converges, and a small protocol change to fix

[David Capwell]

> Won't the replacement have a newer generation?

The replacement is a different instance.  I performs a shadow round with its 
seeds and if they are impacted by this issue then they are missing tokens, so 
we fail the host replacement… you can work around this by changing the seeds to 
nodes that know the token.

> I don't think it is, this is just fixing a gossip bug, and we should do so in 
> all affected versions.

Right now we sent NoPayload which is 0 bytes, but with the change we send 
GossipShutdown which contains the whole EndpointState… I “feel” like 4.x can 
not handle this but worth a test (after deserializing the message we have extra 
bytes… won’t we get mad about that?)… the gossip handler doesn’t look at the 
payload so as long as 4.x serialization can support this, then it won’t be hard 
to back port to 4.x


> On Oct 6, 2023, at 3:57 PM, Brandon Williams  wrote:
> 
> On Fri, Oct 6, 2023 at 5:50 PM David Capwell  wrote:
>> Lets say you now need to host replace node1
> 
> Won't the replacement have a newer generation?
> 
>> avoid peers mutating endpoint states they don’t own
> 
> This sounds reasonable to me.
> 
>> This would be a protocol change, so would need to make sure everyone is cool 
>> with me doing this in 5.0.>
> 
> I don't think it is, this is just fixing a gossip bug, and we should
> do so in all affected versions.
> 
> Kind Regards,
> Brandon

[DISCUSS] Gossip shutdown may corrupt peers making it so the cluster never converges, and a small protocol change to fix

[David Capwell]

Just filed https://issues.apache.org/jira/browse/CASSANDRA-18913 (Gossip NPE 
due to shutdown event corrupting empty statuses) which is where I saw this 
issue..

When we do gossip shutdown we send a message GOSSIP_SHUTDOWN which then gets 
handled by this method org.apache.cassandra.gms.Gossiper#markAsShutdown… there 
is a issue with the current implementation; the peers mutate the state for the 
node shutting down, which cause pending gossip events to get ignored!

Simple example of an issue here is the following

Node1 starts up and starts bootstrapping
Node1 joins the ring
Node1 disables gossip (or halts)

In this case some nodes in the cluster will see the joining of the ring, and 
others won’t.  Now, the ones who have seen the gossip shutdown will set the 
version to Integer.MAX_VALUE which will have gossip not sync any unseen states…

Why is this a problem?  Lets say you now need to host replace node1… and the 
seeds you are using didn’t see the join ring event… you then get the following 
error during the host replacement "Could not find tokens for %s to replace”

To solve this and clean things up, I would like to send the state from the node 
shutting down and avoid peers mutating endpoint states they don’t own; with 
this the cluster should eventually converge!  This would be a protocol change, 
so would need to make sure everyone is cool with me doing this in 5.0.

Re: [VOTE] Accept java-driver

[David Capwell]

+1

> On Oct 3, 2023, at 8:32 AM, Chris Lohfink  wrote:
> 
> +1
> 
> On Tue, Oct 3, 2023 at 10:30 AM Jeff Jirsa  > wrote:
>> +1
>> 
>> 
>> On Mon, Oct 2, 2023 at 9:53 PM Mick Semb Wever > > wrote:
>>> The donation of the java-driver is ready for its IP Clearance vote.
>>> https://incubator.apache.org/ip-clearance/cassandra-java-driver.html
>>> 
>>> The SGA has been sent to the ASF.  This does not require acknowledgement 
>>> before the vote.
>>> 
>>> Once the vote passes, and the SGA has been filed by the ASF Secretary, we 
>>> will request ASF Infra to move the datastax/java-driver as-is to 
>>> apache/java-driver
>>> 
>>> This means all branches and tags, with all their history, will be kept.  A 
>>> cleaning effort has already cleaned up anything deemed not needed.
>>> 
>>> Background for the donation is found in CEP-8: 
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+DataStax+Drivers+Donation
>>> 
>>> PMC members, please take note of (and check) the IP Clearance requirements 
>>> when voting.
>>> 
>>> The vote will be open for 72 hours (or longer). Votes by PMC members are 
>>> considered binding. A vote passes if there are at least three binding +1s 
>>> and no -1's.
>>> 
>>> regards,
>>> Mick

Re: multiple ParameterizedClass objects?

[David Capwell]

It would help me if you could give examples of what you want the yaml to look 
like and why it requires ParameterizedClass.  I try to avoid that class as much 
as possible when doing configs and newer configs are finding different ways to 
solve the same problems...

> On Oct 3, 2023, at 12:10 AM, Claude Warren, Jr via dev 
>  wrote:
> 
> I have a case where I would like to be able to specify a collection of 
> ParameterizedClass objects in the configuration file.  Is there a standard 
> way to do this?  If not, does anyone have a suggestion for a clean way to 
> implement it?
> 
> Claude

Re: [DISCUSS] Backport CASSANDRA-18816 to 5.0? Add support for repair coordinator to retry messages that timeout

[David Capwell]

Thanks all for the feedback!  The patch has 2 +1s on trunk and back ported to 
5.0, making sure it’s stable now; I plan to merge early this week.

> On Sep 21, 2023, at 2:07 PM, Ekaterina Dimitrova  
> wrote:
> 
> +1 from me too. Moreover, this work has started as part of the test efforts 
> and identifying weak points during the 4.0 testing, if I recall correctly. 
> 5.0 sounds like a good place to land. Thank you David and everyone else 
> involved for your efforts!
> 
> On Thu, 21 Sep 2023 at 1:01, Berenguer Blasi  <mailto:berenguerbl...@gmail.com>> wrote:
>> +1 I agree with Brandon. It's more like a bug imo.
>> On 20/9/23 21:42, Caleb Rackliffe wrote:
>>> +1 on a 5.0 backport
>>> 
>>> On Wed, Sep 20, 2023 at 2:26 PM Brandon Williams >> <mailto:dri...@gmail.com>> wrote:
>>>> I think it could be argued that not retrying messages is a bug, I am
>>>> +1 on including this in 5.0.
>>>> 
>>>> Kind Regards,
>>>> Brandon
>>>> 
>>>> On Tue, Sep 19, 2023 at 1:16 PM David Capwell >>> <mailto:dcapw...@apple.com>> wrote:
>>>> >
>>>> > To try to get repair more stable, I added optional retry logic (patch is 
>>>> > still in review) to a handful of critical repair verbs.  This patch is 
>>>> > disabled by default but allows you to opt-in to retries so ephemeral 
>>>> > issues don’t cause a repair to fail after running for a long time 
>>>> > (assuming they resolve within the retry window). There are 2 protocol 
>>>> > level changes to enable this: VALIDATION_RSP and SYNC_RSP now send an 
>>>> > ACK (if the sender doesn’t attach a callback, these ACKs get ignored in 
>>>> > all versions; see org.apache.cassandra.net 
>>>> > <http://org.apache.cassandra.net/>.ResponseVerbHandler#doVerb and 
>>>> > Verb.REPAIR_RSP).  Given that we have already forked, I believe we would 
>>>> > need to give a waiver to allow this patch due to this change.
>>>> >
>>>> > The patch was written on trunk, but figured back porting 5.0 would be 
>>>> > rather trivial and this was brought up during the review, so floating 
>>>> > this to a wider audience.
>>>> >
>>>> > If you look at the patch you will see that it is very large, but this is 
>>>> > only to make testing of repair coordination easier and deterministic, 
>>>> > the biggest code changes are:
>>>> >
>>>> > 1) Moving from ActiveRepairService.instance to 
>>>> > ActiveRepairService.instance() (this is the main reason so many files 
>>>> > were touched; this was needed so unit tests don’t load the whole world)
>>>> > 2) Repair no longer reaches into global space and instead is provided 
>>>> > the subsystems needed to perform repair; this change is local to repair 
>>>> > code
>>>> >
>>>> > Both of these changes were only for testing as they allow us to simulate 
>>>> > 1k repairs in around 15 seconds with 100% deterministic execution.

Re: [DISCUSS] Vector type and empty value

[David Capwell]

> I don’t think we can readily migrate old types away from this however, 
> without breaking backwards compatibility. 

Given that java driver has a different behavior from server, I wouldn’t be 
shocked to see that other drivers also have their own custom behaviors… so not 
clear how to migrate unless we actually hand a user facing standard per type… 
if all drivers use a “default value” and is consistent, I do think we could 
migrate, but would need to live with this till at least 6.0+

> We can only prevent its use in the CQL layer where support isn’t required.

+1

> On Sep 20, 2023, at 7:38 AM, Benedict  wrote:
> 
> Yes, if this is what was meant by empty I agree. It’s nonsensical for most 
> types. Apologies for any confusion.
> 
> I don’t think we can readily migrate old types away from this however, 
> without breaking backwards compatibility. We can only prevent its use in the 
> CQL layer where support isn’t required. My understanding was that we had at 
> least tried to do this for all non-thrift schemas, but perhaps we did not do 
> so thoroughly and now may have some CQL legacy support requirements as well.
> 
>> On 20 Sep 2023, at 15:30, Aleksey Yeshchenko  wrote:
>> 
>> Allowing zero-length byte arrays for most old types is just a legacy from 
>> Darker Days. It’s a distinct concern from columns being nullable or not.
>> 
>> There are a couple types where this makes sense: strings and blobs. All else 
>> should not allow this except for backward compatibility reasons. So, not for 
>> new types.
>> 
>>>> On 20 Sep 2023, at 00:08, David Capwell  wrote:
>>>> 
>>>> When does empty mean null?
>>> 
>>> 
>>> Most types are this way
>>> 
>>> @Test
>>> public void nullExample()
>>> {
>>> createTable("CREATE TABLE %s (pk int primary key, cuteness int)");
>>> execute("INSERT INTO %s (pk, cuteness) VALUES (0, ?)", ByteBuffer.wrap(new 
>>> byte[0]));
>>> Row result = execute("SELECT * FROM %s WHERE pk=0").one();
>>> if (result.has("cuteness")) System.out.println("Cuteness score: " + 
>>> result.getInt("cuteness"));
>>> else System.out.println("Cuteness score is undefined");
>>> }
>>> 
>>> 
>>> This test will NPE in getInt as the returned BB is seen as “null” for int32 
>>> type, you can make it “safer” by changing to the following
>>> 
>>> if (result.has("cuteness")) System.out.println("Cuteness score: " + 
>>> Int32Type.instance.compose(result.getBlob("cuteness")));
>>> 
>>> Now we get the log "Cuteness score: null”
>>> 
>>> What’s even better (just found this out) is that client isn’t consistent or 
>>> correct in these cases!
>>> 
>>> com.datastax.driver.core.Row result = executeNet(ProtocolVersion.CURRENT, 
>>> "SELECT * FROM %s WHERE pk=0").one();
>>> if (result.getBytesUnsafe("cuteness") != null) System.out.println("Cuteness 
>>> score: " + result.getInt("cuteness"));
>>> else System.out.println("Cuteness score is undefined”);
>>> 
>>> This prints "Cuteness score: 0”
>>> 
>>> So for Cassandra we think the value is “null” but java driver thinks it’s 0?
>>> 
>>>> Do we have types where writing an empty value creates a tombstone?
>>> 
>>> Empty does not generate a tombstone for any type, but empty has a similar 
>>> user experience as we return null in both cases (but just found out that 
>>> the drivers may not be consistent with this…)
>>> 
>>>>> On Sep 19, 2023, at 3:33 PM, J. D. Jordan  
>>>>> wrote:
>>>> 
>>>> When does empty mean null?  My understanding was that empty is a valid 
>>>> value for the types that support it, separate from null (aka a tombstone). 
>>>> Do we have types where writing an empty value creates a tombstone?
>>>> 
>>>> I agree with David that my preference would be for only blob and string 
>>>> like types to support empty. It’s too late for the existing types, but we 
>>>> should hold to this going forward. Which is what I think the idea was in 
>>>> https://issues.apache.org/jira/browse/CASSANDRA-8951 as well?  That it was 
>>>> sad the existing numerics were emptiable, but too late to change, and we 
>>>> could correct it for newer types.
>>>> 
>>>>> On Sep 19, 2023, at 12:12 PM, David Capwell  wrote:
>

Re: [DISCUSS] Vector type and empty value

[David Capwell]

> So, not for new types.

> Should we make the Vector type non-emptiable and stick to it for the new 
> types?

Yep, works for me.

We should also update the test 
org.apache.cassandra.db.marshal.AbstractTypeTest#empty to detect this for new 
types by making org.apache.cassandra.db.marshal.AbstractType#allowsEmpty 
default to false and override in all legacy types

More than glad to review any patch to fix this issue!

> On Sep 20, 2023, at 7:16 AM, Aleksey Yeshchenko  wrote:
> 
> Allowing zero-length byte arrays for most old types is just a legacy from 
> Darker Days. It’s a distinct concern from columns being nullable or not.
> 
> There are a couple types where this makes sense: strings and blobs. All else 
> should not allow this except for backward compatibility reasons. So, not for 
> new types.
> 
>> On 20 Sep 2023, at 00:08, David Capwell  wrote:
>> 
>>> When does empty mean null?
>> 
>> 
>> Most types are this way
>> 
>> @Test
>> public void nullExample()
>> {
>> createTable("CREATE TABLE %s (pk int primary key, cuteness int)");
>> execute("INSERT INTO %s (pk, cuteness) VALUES (0, ?)", ByteBuffer.wrap(new 
>> byte[0]));
>> Row result = execute("SELECT * FROM %s WHERE pk=0").one();
>> if (result.has("cuteness")) System.out.println("Cuteness score: " + 
>> result.getInt("cuteness"));
>> else System.out.println("Cuteness score is undefined");
>> }
>> 
>> 
>> This test will NPE in getInt as the returned BB is seen as “null” for int32 
>> type, you can make it “safer” by changing to the following
>> 
>> if (result.has("cuteness")) System.out.println("Cuteness score: " + 
>> Int32Type.instance.compose(result.getBlob("cuteness")));
>> 
>> Now we get the log "Cuteness score: null”
>> 
>> What’s even better (just found this out) is that client isn’t consistent or 
>> correct in these cases!
>> 
>> com.datastax.driver.core.Row result = executeNet(ProtocolVersion.CURRENT, 
>> "SELECT * FROM %s WHERE pk=0").one();
>> if (result.getBytesUnsafe("cuteness") != null) System.out.println("Cuteness 
>> score: " + result.getInt("cuteness"));
>> else System.out.println("Cuteness score is undefined”);
>> 
>> This prints "Cuteness score: 0”
>> 
>> So for Cassandra we think the value is “null” but java driver thinks it’s 0?
>> 
>>> Do we have types where writing an empty value creates a tombstone?
>> 
>> Empty does not generate a tombstone for any type, but empty has a similar 
>> user experience as we return null in both cases (but just found out that the 
>> drivers may not be consistent with this…)
>> 
>>> On Sep 19, 2023, at 3:33 PM, J. D. Jordan  wrote:
>>> 
>>> When does empty mean null?  My understanding was that empty is a valid 
>>> value for the types that support it, separate from null (aka a tombstone). 
>>> Do we have types where writing an empty value creates a tombstone?
>>> 
>>> I agree with David that my preference would be for only blob and string 
>>> like types to support empty. It’s too late for the existing types, but we 
>>> should hold to this going forward. Which is what I think the idea was in 
>>> https://issues.apache.org/jira/browse/CASSANDRA-8951 as well?  That it was 
>>> sad the existing numerics were emptiable, but too late to change, and we 
>>> could correct it for newer types.
>>> 
>>>> On Sep 19, 2023, at 12:12 PM, David Capwell  wrote:
>>>> 
>>>> 
>>>>> 
>>>>> When we introduced TINYINT and SMALLINT (CASSANDRA-8951) we started 
>>>>> making types non -emptiable. This approach makes more sense to me as 
>>>>> having to deal with empty value is error prone in my opinion.
>>>> 
>>>> I agree it’s confusing, and in the patch I found that different code paths 
>>>> didn’t handle things correctly as we have some times (most) that support 
>>>> empty bytes, and some that do not…. Empty also has different meaning in 
>>>> different code paths; for most it means “null”, and for some other types 
>>>> it means “empty”…. To try to make things more clear I added 
>>>> org.apache.cassandra.db.marshal.AbstractType#isNull(V, 
>>>> org.apache.cassandra.db.marshal.ValueAccessor) to the type system so 
>>>> each type can define if empty is null or not.
>>>> 
>>>>> I also think 

Re: [DISCUSS] Vector type and empty value

[David Capwell]

> When does empty mean null?


Most types are this way

@Test
public void nullExample()
{
createTable("CREATE TABLE %s (pk int primary key, cuteness int)");
execute("INSERT INTO %s (pk, cuteness) VALUES (0, ?)", ByteBuffer.wrap(new 
byte[0]));
Row result = execute("SELECT * FROM %s WHERE pk=0").one();
if (result.has("cuteness")) System.out.println("Cuteness score: " + 
result.getInt("cuteness"));
else System.out.println("Cuteness score is undefined");
}


This test will NPE in getInt as the returned BB is seen as “null” for int32 
type, you can make it “safer” by changing to the following

if (result.has("cuteness")) System.out.println("Cuteness score: " + 
Int32Type.instance.compose(result.getBlob("cuteness")));

Now we get the log "Cuteness score: null”

What’s even better (just found this out) is that client isn’t consistent or 
correct in these cases!

com.datastax.driver.core.Row result = executeNet(ProtocolVersion.CURRENT, 
"SELECT * FROM %s WHERE pk=0").one();
if (result.getBytesUnsafe("cuteness") != null) System.out.println("Cuteness 
score: " + result.getInt("cuteness"));
else System.out.println("Cuteness score is undefined”);

This prints "Cuteness score: 0”

So for Cassandra we think the value is “null” but java driver thinks it’s 0?

> Do we have types where writing an empty value creates a tombstone?

Empty does not generate a tombstone for any type, but empty has a similar user 
experience as we return null in both cases (but just found out that the drivers 
may not be consistent with this…)

> On Sep 19, 2023, at 3:33 PM, J. D. Jordan  wrote:
> 
> When does empty mean null?  My understanding was that empty is a valid value 
> for the types that support it, separate from null (aka a tombstone). Do we 
> have types where writing an empty value creates a tombstone?
> 
> I agree with David that my preference would be for only blob and string like 
> types to support empty. It’s too late for the existing types, but we should 
> hold to this going forward. Which is what I think the idea was in 
> https://issues.apache.org/jira/browse/CASSANDRA-8951 as well?  That it was 
> sad the existing numerics were emptiable, but too late to change, and we 
> could correct it for newer types.
> 
>> On Sep 19, 2023, at 12:12 PM, David Capwell  wrote:
>> 
>> 
>>> 
>>> When we introduced TINYINT and SMALLINT (CASSANDRA-8951) we started making 
>>> types non -emptiable. This approach makes more sense to me as having to 
>>> deal with empty value is error prone in my opinion.
>> 
>> I agree it’s confusing, and in the patch I found that different code paths 
>> didn’t handle things correctly as we have some times (most) that support 
>> empty bytes, and some that do not…. Empty also has different meaning in 
>> different code paths; for most it means “null”, and for some other types it 
>> means “empty”…. To try to make things more clear I added 
>> org.apache.cassandra.db.marshal.AbstractType#isNull(V, 
>> org.apache.cassandra.db.marshal.ValueAccessor) to the type system so each 
>> type can define if empty is null or not.
>> 
>>> I also think that it would be good to standardize on one approach to avoid 
>>> confusion.
>> 
>> I agree, but also don’t feel it’s a perfect one-size-fits-all thing…. Let’s 
>> say I have a “blob” type and I write an empty byte… what does this mean?  
>> What does it mean for "text" type?  The fact I get back a null in both those 
>> cases was very confusing to me… I do feel that some types should support 
>> empty, and the common code of empty == null I think is very brittle 
>> (blob/text was not correct in different places due to this...)… so I am cool 
>> with removing that relationship, but don’t think we should have a rule 
>> blocking empty for all current / future types as it some times does make 
>> sense.
>> 
>>> empty vector (I presume) for the vector type?
>> 
>> Empty vectors (vector[0]) are blocked at the type level, the smallest vector 
>> is vector[1]
>> 
>>> as types that can never be null
>> 
>> One pro here is that “null” is cheaper (in some regards) than delete (though 
>> we can never purge), but having 2 similar behaviors (write null, do a 
>> delete) at the type level is a bit confusing… Right now I am allowed to do 
>> the following (the below isn’t valid CQL, its a hybrid of CQL + Java code…)
>> 
>> CREATE TABLE fluffykittens (pk int primary key, cuteness int);
>> INSERT INTO fluffykittens (pk, cuteness) VALUES (0, new byte[0])
>> 
>> CREATE TABLE typesareh

[DISCUSS] Backport CASSANDRA-18816 to 5.0? Add support for repair coordinator to retry messages that timeout

[David Capwell]

To try to get repair more stable, I added optional retry logic (patch is still 
in review) to a handful of critical repair verbs.  This patch is disabled by 
default but allows you to opt-in to retries so ephemeral issues don’t cause a 
repair to fail after running for a long time (assuming they resolve within the 
retry window). There are 2 protocol level changes to enable this: 
VALIDATION_RSP and SYNC_RSP now send an ACK (if the sender doesn’t attach a 
callback, these ACKs get ignored in all versions; see 
org.apache.cassandra.net.ResponseVerbHandler#doVerb and Verb.REPAIR_RSP).  
Given that we have already forked, I believe we would need to give a waiver to 
allow this patch due to this change.

The patch was written on trunk, but figured back porting 5.0 would be rather 
trivial and this was brought up during the review, so floating this to a wider 
audience.

If you look at the patch you will see that it is very large, but this is only 
to make testing of repair coordination easier and deterministic, the biggest 
code changes are:

1) Moving from ActiveRepairService.instance to ActiveRepairService.instance() 
(this is the main reason so many files were touched; this was needed so unit 
tests don’t load the whole world)
2) Repair no longer reaches into global space and instead is provided the 
subsystems needed to perform repair; this change is local to repair code

Both of these changes were only for testing as they allow us to simulate 1k 
repairs in around 15 seconds with 100% deterministic execution.

Re: [DISCUSS] Vector type and empty value

[David Capwell]

> When we introduced TINYINT and SMALLINT (CASSANDRA-895) we started making 
> types non -emptiable. This approach makes more sense to me as having to deal 
> with empty value is error prone in my opinion.

I agree it’s confusing, and in the patch I found that different code paths 
didn’t handle things correctly as we have some times (most) that support empty 
bytes, and some that do not…. Empty also has different meaning in different 
code paths; for most it means “null”, and for some other types it means 
“empty”…. To try to make things more clear I added 
org.apache.cassandra.db.marshal.AbstractType#isNull(V, 
org.apache.cassandra.db.marshal.ValueAccessor) to the type system so each 
type can define if empty is null or not.

> I also think that it would be good to standardize on one approach to avoid 
> confusion.

I agree, but also don’t feel it’s a perfect one-size-fits-all thing…. Let’s say 
I have a “blob” type and I write an empty byte… what does this mean?  What does 
it mean for "text" type?  The fact I get back a null in both those cases was 
very confusing to me… I do feel that some types should support empty, and the 
common code of empty == null I think is very brittle (blob/text was not correct 
in different places due to this...)… so I am cool with removing that 
relationship, but don’t think we should have a rule blocking empty for all 
current / future types as it some times does make sense.

> empty vector (I presume) for the vector type?

Empty vectors (vector[0]) are blocked at the type level, the smallest vector is 
vector[1]

>  as types that can never be null

One pro here is that “null” is cheaper (in some regards) than delete (though we 
can never purge), but having 2 similar behaviors (write null, do a delete) at 
the type level is a bit confusing… Right now I am allowed to do the following 
(the below isn’t valid CQL, its a hybrid of CQL + Java code…)

CREATE TABLE fluffykittens (pk int primary key, cuteness int);
INSERT INTO fluffykittens (pk, cuteness) VALUES (0, new byte[0])

CREATE TABLE typesarehard (pk1 int, pk2 int, cuteness int, PRIMARY KEY ((pk1, 
pk2));
INSERT INTO typesarehard (pk1, pk2, cuteness) VALUES (new byte[0], new byte[0], 
new byte[0]) — valid as the partition key is not empty as its a composite of 2 
empty values, this is the same as new byte[2]

The first time I ever found out that empty bytes was valid was when a user was 
trying to abuse this in collections (also the fact collections support null in 
some cases and not others is fun…)…. It was blowing up in random places… good 
times!

I am personally not in favor of allowing empty bytes (other than for blob / 
text as that is actually valid for the domain), but having similar types having 
different semantics I feel is more problematic...

> On Sep 19, 2023, at 8:56 AM, Josh McKenzie  wrote:
> 
>> I am strongly in favour of permitting the table definition forbidding nulls 
>> - and perhaps even defaulting to this behaviour. But I don’t think we should 
>> have types that are inherently incapable of being null.
> I'm with Benedict. Seems like this could help prevent whatever "nulls in 
> primary key columns" problems Aleksey was alluding to on those tickets back 
> in the day that pushed us towards making the new types non-emptiable as well 
> (i.e. primary keys are non-null in table definition).
> 
> Furthering Alex' question, having a default value for unset fields in any 
> non-collection context seems... quite surprising to me in a database. I could 
> see the argument for making container / collection types non-nullable, maybe, 
> but that just keeps us in a potential straddle case (some types nullable, 
> some not).
> 
> On Tue, Sep 19, 2023, at 8:22 AM, Benedict wrote:
>> 
>> If I understand this suggestion correctly it is a whole can of worms, as 
>> types that can never be null prevent us ever supporting outer joins that 
>> return these types.
>> 
>> I am strongly in favour of permitting the table definition forbidding nulls 
>> - and perhaps even defaulting to this behaviour. But I don’t think we should 
>> have types that are inherently incapable of being null. I also certainly 
>> don’t think we should have bifurcated our behaviour between types like this.
>> 
>> 
>> 
>>> On 19 Sep 2023, at 11:54, Alex Petrov  wrote:
>>> 
>>> To make sure I understand this right; does that mean there will be a 
>>> default value for unset fields? Like 0 for numerical values, and an empty 
>>> vector (I presume) for the vector type?
>>> 
>>> On Fri, Sep 15, 2023, at 11:46 AM, Benjamin Lerer wrote:
 Hi everybody,
 
 I noticed that the new Vector type accepts empty ByteBuffer values as an 
 input representing null.
 When we introduced TINYINT and SMALLINT (CASSANDRA-895) we started making 
 types non -emptiable. This approach makes more sense to me as having to 
 deal with empty value is error prone in my opinion.
 I also think that it would be good to standardize on one 

[DISCUSS] Add Jepsen's Elle as a test dependency for Accord / Paxos

[David Capwell]

For validation of Paxos and Accord 2 different consistency verifiers were 
created: accord.verify.StrictSerializabilityVerifier (Accord), and 
org.apache.cassandra.simulator.paxos.LinearizabilityValidator (Paxos).  To 
increase confidence in both protocols it would be good to use an external 
consistency checker, such as Jepsen's Elle.

This work was first started in 
https://the-asf.slack.com/archives/C0459N9R5C6/p1692192925909199 by Jarek, and 
would be good to get as part of our automation.


https://github.com/jepsen-io/elle/blob/main/LICENSE - Eclipse Public License 2.0

Re: [Discuss] Repair inside C*

[David Capwell]

+0 to sidecar, in order to make that work well we need to expose state that the 
node has so the sidecar can make good calls, if it runs in the node then 
nothing has to be exposed.  One thing to flesh out is where do the “smarts” 
live?  If the range has too many partitions, which system knows to subdivide 
the range and sequence the repairs (else you OOM)?  “Should” repair itself be 
better and take all input and make sure it works correctly, so the caller just 
worries about scheduling?  “Should” the scheduler understand limitations with 
repair and work around them?

> On Jul 25, 2023, at 11:26 AM, Jeremiah Jordan  
> wrote:
> 
> +1 for the side car being the right location.
> 
> -Jeremiah
> 
> On Jul 25, 2023 at 1:16:14 PM, Chris Lohfink  <mailto:clohfin...@gmail.com>> wrote:
>> I think a CEP is the next step. Considering the number of companies 
>> involved, this might necessitate several drafts and rounds of discussions. I 
>> appreciate your initiative in starting this process, and I'm eager to 
>> contribute to the ensuing discussions. Maybe in a google docs or something 
>> initially for more interactive feedback?
>> 
>> In regards to https://issues.apache.org/jira/browse/CASSANDRA-14346 we at 
>> Netflix are actually putting effort currently to move this into the sidecar 
>> as the idea was to start moving non-read/write path things into different 
>> process and jvms to not impact each other.
>> 
>> I think the sidecar/in process discussion might be a bit contentious as I 
>> know even things like compaction some feel should be moved out of process in 
>> future. On a personal note, my primary interest lies in seeing the 
>> implementation realized, so I am willing to support whatever consensus 
>> emerges. Whichever direction these go we will help with the implementation.
>> 
>> Chris
>> 
>> On Tue, Jul 25, 2023 at 1:09 PM Jaydeep Chovatia > <mailto:chovatia.jayd...@gmail.com>> wrote:
>>> Sounds good, German. Feel free to let me know if you need my help in filing 
>>> CEP, adding supporting content to the CEP, etc.
>>> As I mentioned previously, I have already been working (going through an 
>>> internal review) on creating a one-pager doc, code, etc., that has been 
>>> working for us for the last six years at an immense scale, and I will share 
>>> it soon on a private fork.
>>> 
>>> Thanks,
>>> Jaydeep
>>> 
>>> On Tue, Jul 25, 2023 at 9:48 AM German Eichberger via dev 
>>> mailto:dev@cassandra.apache.org>> wrote:
>>>> In [2] we suggested that the next step should be a CEP.
>>>> 
>>>> I am happy to lend a hand to this effort as well.
>>>> 
>>>> Thanks Jaydeep and David - really appreciated.
>>>> 
>>>> German
>>>> 
>>>> From: David Capwell mailto:dcapw...@apple.com>>
>>>> Sent: Tuesday, July 25, 2023 8:32 AM
>>>> To: dev mailto:dev@cassandra.apache.org>>
>>>> Cc: German Eichberger >>> <mailto:german.eichber...@microsoft.com>>
>>>> Subject: [EXTERNAL] Re: [Discuss] Repair inside C*
>>>>  
>>>> As someone who has done a lot of work trying to make repair stable, I 
>>>> approve of this message ^_^
>>>> 
>>>> More than glad to help mentor this work
>>>> 
>>>> On Jul 24, 2023, at 6:29 PM, Jaydeep Chovatia >>> <mailto:chovatia.jayd...@gmail.com>> wrote:
>>>> 
>>>> To clarify the repair solution timing, the one we have listed in the 
>>>> article is not the recently developed one. We were hitting some 
>>>> high-priority production challenges back in early 2018, and to address 
>>>> that, we developed and rolled out the solution in production in just a few 
>>>> months. The timing-wise, the solution was developed and productized by Q3 
>>>> 2018, of course, continued to evolve thereafter. Usually, we explore the 
>>>> existing solutions we can leverage, but when we started our journey in 
>>>> early 2018, most of the solutions were based on sidecar solutions. There 
>>>> is nothing against the sidecar solution; it was just a pure business 
>>>> decision, and in that, we wanted to avoid the sidecar to avoid a 
>>>> dependency on the control plane. Every solution developed has its deep 
>>>> context, merits, and pros and cons; they are all great solutions! 
>>>> 
>>>> An appeal to the community members is to think one more time about having 
>>>> re

Re: [Discuss] Repair inside C*

[David Capwell]

As someone who has done a lot of work trying to make repair stable, I approve 
of this message ^_^

More than glad to help mentor this work

> On Jul 24, 2023, at 6:29 PM, Jaydeep Chovatia  
> wrote:
> 
> To clarify the repair solution timing, the one we have listed in the article 
> is not the recently developed one. We were hitting some high-priority 
> production challenges back in early 2018, and to address that, we developed 
> and rolled out the solution in production in just a few months. The 
> timing-wise, the solution was developed and productized by Q3 2018, of 
> course, continued to evolve thereafter. Usually, we explore the existing 
> solutions we can leverage, but when we started our journey in early 2018, 
> most of the solutions were based on sidecar solutions. There is nothing 
> against the sidecar solution; it was just a pure business decision, and in 
> that, we wanted to avoid the sidecar to avoid a dependency on the control 
> plane. Every solution developed has its deep context, merits, and pros and 
> cons; they are all great solutions! 
> 
> An appeal to the community members is to think one more time about having 
> repairs in the Open Source Cassandra itself. As mentioned in my previous 
> email, any solution getting adopted is fine; the important aspect is to have 
> a repair solution in the OSS Cassandra itself!
> 
> Yours Faithfully,
> Jaydeep
> 
> On Mon, Jul 24, 2023 at 3:46 PM Jaydeep Chovatia  > wrote:
>> Hi German,
>> 
>> The goal is always to backport our learnings back to the community. For 
>> example, I have already successfully backported the following two 
>> enhancements/bug fixes back to the Open Source Cassandra, which are 
>> described in the article. I am already currently working on open-source a 
>> few more enhancements mentioned in the article back to the open-source.
>> https://issues.apache.org/jira/browse/CASSANDRA-18555
>> https://issues.apache.org/jira/browse/CASSANDRA-13740
>> There is definitely heavy interest in having the repair solution inside the 
>> Open Source Cassandra itself, very much like Compaction. As I write this 
>> email, we are internally working on a one-pager proposal doc to all the 
>> community members on having a repair inside the OSS Apache Cassandra along 
>> with our private fork - I will share it soon.
>> 
>> Generally, we are ok with any solution getting adopted (either Joey's 
>> solution or our repair solution or any other solution). The primary 
>> motivation is to have the repair embedded inside the open-source Cassandra 
>> itself, so we can retire all various privately developed solutions 
>> eventually :)
>> 
>> I am also happy to help (drive conversation, discussion, etc.) in any way to 
>> have a repair solution adopted inside Cassandra itself, please let me know. 
>> Happy to help!
>> 
>> Yours Faithfully,
>> Jaydeep
>> 
>> On Mon, Jul 24, 2023 at 1:44 PM German Eichberger via dev 
>> mailto:dev@cassandra.apache.org>> wrote:
>>> All,
>>> 
>>> We had a brief discussion in [2] about the Uber article [1] where they talk 
>>> about having integrated repair into Cassandra and how great that is. I 
>>> expressed my disappointment that they didn't work with the community on 
>>> that (Uber, if you are listening time to make amends ) and it turns out 
>>> Joey already had the idea and wrote the code [3] - so I wanted to start a 
>>> discussion to gauge interest and maybe how to revive that effort.
>>> 
>>> Thanks,
>>> German
>>> 
>>> [1] 
>>> https://www.uber.com/blog/how-uber-optimized-cassandra-operations-at-scale/
>>> [2] https://the-asf.slack.com/archives/CK23JSY2K/p1690225062383619
>>> [3] https://issues.apache.org/jira/browse/CASSANDRA-14346

Re: [DISCUSS] When to run CheckStyle and other verificiations

[David Capwell]

> nobody referred to running checks in a pre-push (or pre-commit) hook


In accord I added an opt-out for each hook, and will require such here as well… 
as long as you can opt-out, its fine by me… I know I will likely opt-out, but 
wouldn’t block such an effort

> Your point that pre-push hook might not be the best one is valid, and we 
> should rather think about pre-commit

Pre-push is far better than pre-commit, with pre-commit you are forcing a style 
on people…. I for one have many many commits in a single PR, where I use 
commits to not loose track of progress (even if the code doesn’t compile), so 
forcing the build to work would be a -1 from me…. Pre-push at least means “you 
want the world to see this” so makes more sense there…

Again, must have an opt-out

> proposed:
> ant jar (just build)
> git commit (run some checks)

I am against this, jar should also check and ask users to opt-out if they don’t 
want the checks….

> If we go with opt-out i'd like to see one flag that can disable all checks: 
> `-Dchecks.skip`

Works for me, you can also do the following to disable and not worry about

$ cat < build.properties
checks.skip: true
EOF

> On Jun 27, 2023, at 3:14 AM, Mick Semb Wever  wrote:
> 
>> The context is that we currently have 3 checks in the build:
>> 
>> - Checkstyle,
>> - Eclipse-Warnings,
>> - RAT
> 
> 
> And dependency-check (owasp).
> 
> 
> 
>> I want to discuss whether you are ok with extracting all checks to their 
>> distinct target and not running it automatically with the targets which devs 
>> usually run locally. In particular:
>> 
>> 
>> "build", "jar", and all "test" targets would not trigger CheckStyle, RAT or 
>> Eclipse-Warnings
>> A new target "check" would trigger all CheckStyle, RAT, and Eclipse-Warnings
>> The new "check" target would be run along with the "artifacts" target on 
>> Jenkins-CI, and it as a separate build step in CircleCI
> 
> 
> +0 I prefer this opt-in over an opt-out approach.
> 
> It should be separated from `artifacts` too.
> We would need to encourage engineers to run `ant check` before
> starting CI and/or requesting review.
> 
> I'm in favour of the opt-in approach because it keeps it visible.
> Folks configure flags and it "disappears" forever.  Also it's a
> headache in all the other ant targets where we actually don't want it,
> e.g. tests.
> 
> If we go with opt-out i'd like to see one flag that can disable all
> checks: `-Dchecks.skip`
> 
> 
>> That could be fixed by running checks in a pre-push Git hook. There are some 
>> benefits of this compared to the current behavior:
> 
> 
> -1
> committing and pushing to a personal branch is often done to save work
> or for cross-machine or collaboration. We should not gate on checks or
> compilation here.
> 
> PRs should fail if checks fail, to give newcomers clear feedback (and
> to take this nit-picking out of the review process).

Re: [DISCUSS] When to run CheckStyle and other verificiations

[David Capwell]

> not running it automatically with the targets which devs usually run locally.

The checks tend to have an opt-out, such as -Dno-checkstyle=true… so its really 
easy to setup your local environment to opt out what you do not care about… I 
feel we should force people to opt-out rather than opt-in… 



> On Jun 26, 2023, at 7:47 AM, Jacek Lewandowski  
> wrote:
> 
> That would work as well Brandon, basically what is proposed in 
> CASSANDRA-18618, that is "check" target, actually needs to build the project 
> to perform some verifications - I suppose running "ant check" should be 
> sufficient.  
> 
> - - -- --- -  -
> Jacek Lewandowski
> 
> 
> pon., 26 cze 2023 o 16:01 Brandon Williams  > napisał(a):
>> The "artifacts" task is not quite the same since it builds other things like 
>> docs, which significantly contributes to longer build time.  I don't see why 
>> we couldn't add a new task that preserves the old behavior though, "fulljar" 
>> or something like that.
>> 
>> Kind Regards,
>> Brandon
>> 
>> 
>> On Mon, Jun 26, 2023 at 6:12 AM Jacek Lewandowski 
>> mailto:lewandowski.ja...@gmail.com>> wrote:
>>> Yes, I've mentioned that there is a property we can set to skip checkstyle.
>>> 
>>> Currently such a goal is "artifacts" which basically validates everything.
>>> 
>>> 
>>> - - -- --- -  -
>>> Jacek Lewandowski
>>> 
>>> 
>>> pon., 26 cze 2023 o 13:09 Mike Adamson >> > napisał(a):
 While I like the idea of this because of added time these checks take, I 
 was under the impression that checkstyle (at least) can be disabled with a 
 flag. 
 
 If we did do this, would it make sense to have a "release"  or "commit" 
 target (or some other name) that ran a full build with all checks that can 
 be used prior to pushing changes?
 
 On Mon, 26 Jun 2023 at 08:35, Berenguer Blasi >>> > wrote:
> I would prefer sthg that is totally transparent to me and not add one 
> more step I have to remember. Just to push/run CI to find out I missed it 
> and rinse and repeat... With the recent fix to checkstyle I am happy as 
> things stand atm. My 2cts
> 
> On 26/6/23 8:43, Jacek Lewandowski wrote:
>> Hi,
>> 
>> The context is that we currently have 3 checks in the build:
>> - Checkstyle,
>> - Eclipse-Warnings,
>> - RAT
>> 
>> CheckStyle and RAT are executed with almost every target we run: build, 
>> jar, test, test-some, testclasslist, etc.; on the other hand, 
>> Eclipse-Warnings is executed automatically only with the artifacts 
>> target. 
>> 
>> Checkstyle currently uses some caching, so subsequent reruns without 
>> cleaning the project validate only the modified files.
>> 
>> Both CI - Jenkins and Circle forces running all checks.
>> 
>> I want to discuss whether you are ok with extracting all checks to their 
>> distinct target and not running it automatically with the targets which 
>> devs usually run locally. In particular:
>> 
>> "build", "jar", and all "test" targets would not trigger CheckStyle, RAT 
>> or Eclipse-Warnings
>> A new target "check" would trigger all CheckStyle, RAT, and 
>> Eclipse-Warnings
>> The new "check" target would be run along with the "artifacts" target on 
>> Jenkins-CI, and it as a separate build step in CircleCI
>> 
>> The rationale for that change is:
>> Running all the checks together would be more consistent, but running 
>> all of them automatically with build and test targets could waste time 
>> when we develop something locally, frequently rebuilding and running 
>> tests.
>> On the other hand, it would be more consistent if the build did what we 
>> want - as a dev, when prototyping, I don't want to be forced to run 
>> analysis (and potentially fix issues) whenever I want to build a project 
>> or just run a single test.
>> There are ways to avoid running checks automatically by specifying some 
>> build properties. Though, the discussion is about the default behavior - 
>> on the flip side, if one wants to run the checks along with the 
>> specified target, they could add the "check" target to the command line. 
>> 
>> The rationale for keeping the checks running automatically with every 
>> target is to reduce the likelihood of not running the checks locally 
>> before pushing the branch and being surprised by failing CI soon after 
>> starting the build. 
>> 
>> That could be fixed by running checks in a pre-push Git hook. There are 
>> some benefits of this compared to the current behavior:
>> the checks would be run automatically only once
>> they would be triggered even for those devs who do everything in IDE and 
>> do not even touch Ant commands directly
>> 
>> 

Re: [DISCUSS] Using ACCP or tc-native by default

[David Capwell]

+1 to ACCP

> On Jun 22, 2023, at 3:05 PM, C. Scott Andreas  wrote:
> 
> +1 for ACCP and can attest to its results. ACCP also optimizes for a range of 
> hash functions and other cryptographic primitives beyond TLS acceleration for 
> Netty.
> 
>> On Jun 22, 2023, at 2:07 PM, Jeff Jirsa  wrote:
>> 
>> 
>> Either would be better than today. 
>> 
>> On Thu, Jun 22, 2023 at 1:57 PM Jordan West > > wrote:
>>> Hi,
>>> 
>>> I’m wondering if there is appetite to change the default SSL provider for 
>>> Cassandra going forward to either ACCP [1] or tc-native in Netty? Our 
>>> deployment as well as others I’m aware of make this change in their fork 
>>> and it can lead to significant performance improvement. When recently 
>>> qualifying 4.1 without using ACCP (by accident) we noticed p99 latencies 
>>> were 2x higher than 3.0 w/ ACCP. Wiring up ACCP can be a bit of a pain and 
>>> also requires some amount of customization. I think it could be great for 
>>> the wider community to adopt it. 
>>> 
>>> The biggest hurdle I foresee is licensing but ACCP is Apache 2.0 licensed. 
>>> Anything else I am missing before opening a JIRA and submitting a patch?
>>> 
>>> Jordan 
>>> 
>>> 
>>> [1] 
>>> https://github.com/corretto/amazon-corretto-crypto-provider
> 
> 

Re: [DISCUSS] Remove deprecated keyspace_count_warn_threshold and table_count_warn_threshold

[David Capwell]

> Does this sound good?

Sounds good to me

> On Jun 16, 2023, at 8:30 AM, Dan Jatnieks  wrote:
> 
> Hi all,
> 
> Apologies for the late reply; I didn't mean to start a thread and then 
> disappear - it was unintended and I feel bad about that.
> 
> I've been taking notes to summarize the discussion points and it matches what 
> Andres already listed, so I'm glad for that. And thank you Andres for doing 
> that - much appreciated!
> 
> The plan Andres outlined also sounds good to me. I was not aware of @Replaces 
> before, and now that I learned it, I agree it should be used here.
> 
> Converting the old threshold values by subtracting the system keyspace/tables 
> makes sense to keep the existing guardrail semantics - and including a 
> message about that will be a good step to reduce any confusion about how the, 
> possibly odd-looking, new value was determined.
> 
> Dan
> 
> 
> On Fri, Jun 16, 2023 at 9:58 AM Ekaterina Dimitrova  <mailto:e.dimitr...@gmail.com>> wrote:
>> Hi all,
>> I was following the discussion. What Andres just summarized sounds 
>> reasonable to me. Let’s just not forget to document also all this.
>> Thank you
>> Ekaterina
>> 
>> On Fri, 16 Jun 2023 at 10:16, Andrés de la Peña > <mailto:adelap...@apache.org>> wrote:
>>> It seems we agree on removing the default value for the old thresholds, and 
>>> don't count system keyspaces/tables on the new ones.
>>> 
>>> The old thresholds were on active duty for around ten months, and they have 
>>> been deprecated for around a year. They will have been deprecated for 
>>> longer by the time we release 5.0. If we want to keep them in perpetuity, I 
>>> guess the plan would be:
>>> 
>>> - Remove the default value of the old thresholds in Config.java to make 
>>> them disabled by default.
>>> - Remove the old thresholds from the default cassandra.yaml, although old 
>>> yamls can still have them.
>>> - Use converters (@Replaces tag in Config.java) to read the old threshold 
>>> values (if present) and apply them to the new guardrails.
>>> - During the conversion from the old thresholds to the new guardrails, 
>>> subtract the current number of system keyspace/tables from the old value. 
>>> For example, 150 tables in the old threshold translate to 103 tables in the 
>>> new guardrail, considering that there are 47 system tables.
>>> 
>>> Does this sound good?
>>> 
>>> On Wed, 14 Jun 2023 at 17:26, David Capwell >> <mailto:dcapw...@apple.com>> wrote:
>>>>> That's problematic because the new thresholds we added in CASSANDRA-17147 
>>>>> don't include system tables. Do you think we should change that?
>>>> 
>>>> I wouldn’t change the semantics of the config as it’s already live.  I 
>>>> guess where I am coming from is that logically we have to think about the 
>>>> system tables, so to your point, if we think 150 is too much and the 
>>>> system already exposes 50… then we should recommend no more than 100…. 
>>>> 
>>>>> I find it's better for usability to not count the system tables and just 
>>>>> say "It's recommended not to have more than 100 tables. This doesn't 
>>>>> include system tables.”
>>>> 
>>>> 
>>>> I am fine with this framing… internally we think about 150 but publicly 
>>>> speak 100 (due to our 50 tables)...
>>>> 
>>>> 
>>>>> On Jun 14, 2023, at 8:29 AM, Josh McKenzie >>>> <mailto:jmcken...@apache.org>> wrote:
>>>>> 
>>>>>> In my opinion including system tables defeats that purpose because it 
>>>>>> forces users to know details about the system tables.
>>>>> Perhaps having a unit test that caps our system tables at some value and 
>>>>> keeping the guardrail user-scope specific would be a better approach. I 
>>>>> see your point about leaking internal details to users, specifically on 
>>>>> things they can't control at this point.
>>>>> 
>>>>> On Wed, Jun 14, 2023, at 8:19 AM, Andrés de la Peña wrote:
>>>>>> > Default value I agree with you; features should be off by default!  If 
>>>>>> > we remove the default then we disable the feature by default (which im 
>>>>>> > cool with) and for anyone who changed the config, they would keep 
>>>>>> > their behavior
>>>>>> 
>>>>

Re: [DISCUSS] Remove org.apache.cassandra.io.sstable.SSTableHeaderFix in trunk (5.0)?

[David Capwell]

Not heard any feedback yet, so tomorrow plan to remove… the feature was local 
to 3.6+ so all users migrating from 3.0 to 4.0 never had this issue

> On Jun 13, 2023, at 10:22 AM, David Capwell  wrote:
> 
> org.apache.cassandra.io.sstable.SSTableHeaderFix was added due to bugs in 3.6 
> causing invalidate types or incompatible types (due to toString changes) in 
> the SSTableHeader… this logic runs on start and rewrites all Stats files that 
> had a mismatch from the local schema; with 5.0 requiring upgrades from 4.x 
> only, this logic should have already run as its a 3.x to 4.0 migration step 
> (though users are able to opt out [1]) which should have already fixed the 
> SSTables to have correct schema…
> 
> Why is this a problem now?  CASSANDRA-18504 is adding a lot of property/fuzz 
> tests to the type system and the read/write path, which has found several 
> bugs; fixing some of the bugs actually impacts SSTableHeader because it 
> requires generating and working with types that are not valid, so it can fix 
> them…   By removing this logic, we can push this type validation into the 
> type system to avoid generating incorrect types.  
> 
> If we wish to keep this class, we need to maintain allowing invalid types to 
> be created, which may cause bugs down the road.
> 
> 
> [1] if a user opts out there are 2 real cases that are impacted: UDTs, and 
> collections of collections…
> * For UDTs, the frozen vs non-frozen type are not the same, so mixing these 
> causes us to fail to read the data, failing the read…. I believe 
> writes/compactions will not corrupt the data, but anything that touches these 
> SSTables will fail due to the schema mismatch… the only way to resolve this 
> is to fix the SSTables… If you disabled in 4.x, you were living with broken / 
> unreadable SSTables, so by removing 5.0 would loose the ability to repair 
> them (but 4.x would still be able to)
> * for collections of collections, this is less of an issue.  The logic would 
> detect that the collection has a non-frozen collection as the element, so 
> would migrate them to frozen.  This behavior has been moved to the type 
> system, so a read from SSTable of “list>” automagically becomes a 
> "ListType(FrozenType(ListType(Int32Type)))”.  The SSTables are not “fixed”, 
> but compaction is able to read the data correctly, and the new SSTables will 
> have the correct header.  

Re: [DISCUSS] Remove deprecated keyspace_count_warn_threshold and table_count_warn_threshold

[David Capwell]

> That's problematic because the new thresholds we added in CASSANDRA-17147 
> don't include system tables. Do you think we should change that?

I wouldn’t change the semantics of the config as it’s already live.  I guess 
where I am coming from is that logically we have to think about the system 
tables, so to your point, if we think 150 is too much and the system already 
exposes 50… then we should recommend no more than 100…. 

> I find it's better for usability to not count the system tables and just say 
> "It's recommended not to have more than 100 tables. This doesn't include 
> system tables.”


I am fine with this framing… internally we think about 150 but publicly speak 
100 (due to our 50 tables)...


> On Jun 14, 2023, at 8:29 AM, Josh McKenzie  wrote:
> 
>> In my opinion including system tables defeats that purpose because it forces 
>> users to know details about the system tables.
> Perhaps having a unit test that caps our system tables at some value and 
> keeping the guardrail user-scope specific would be a better approach. I see 
> your point about leaking internal details to users, specifically on things 
> they can't control at this point.
> 
> On Wed, Jun 14, 2023, at 8:19 AM, Andrés de la Peña wrote:
>> > Default value I agree with you; features should be off by default!  If we 
>> > remove the default then we disable the feature by default (which im cool 
>> > with) and for anyone who changed the config, they would keep their behavior
>> 
>> I'm glad we agree on at least removing the default value if we keep the 
>> deprecated properties.
>> 
>> > With that, I kinda don’t agree that including system tables is a mistake, 
>> > as we add more we allow less for user tables before we start to have 
>> > issues….
>> 
>> That's problematic because the new thresholds we added in CASSANDRA-17147 
>> don't include system tables. Do you think we should change that?
>> 
>> I still think it's better not to include the system tables in the count. The 
>> thresholds on the number of keyspaces/tables/rows/columns/tombstones are 
>> just guidance since they cannot be exactly related to exact resource 
>> consumption. The main purpose of those thresholds is to prevent obvious 
>> antipatterns such as creating thousands of tables. A benefit of expressing 
>> the guardrails in terms of the number of schema entities, rather than 
>> counting the memory usage of those entities, is that they are easy to 
>> understand and reason about. In my opinion including system tables defeats 
>> that purpose because it forces users to know details about the system 
>> tables. The fact that those details change between versions doesn't help. 
>> Including system tables is not going to make the thresholds precise in terms 
>> of measuring memory consumption because that depends on other factors, such 
>> as the columns they store.
>> 
>> Including system tables also imposes a minimum threshold value, like in 5.0 
>> you cannot set a threshold value under 45 tables without triggering it with 
>> an empty db. For other thresholds, this can be more tricky. That would be 
>> the case of the guardrail on the number of columns in a partition, where you 
>> would need to know the size of the widest row in the system tables, which 
>> can change over time.
>> 
>> I guess that if system tables were to be counted, a recommendation for the 
>> threshold would say something like "It's recommended not to have more than 
>> 150 tables. The system already includes 45 tables for internal usage, so you 
>> shouldn't create more than 105 user tables". I find it's better for 
>> usability to not count the system tables and just say "It's recommended not 
>> to have more than 100 tables. This doesn't include system tables."
>> 
>> On Tue, 13 Jun 2023 at 23:51, Josh McKenzie > <mailto:jmcken...@apache.org>> wrote:
>> 
>>> Warning that too many tables (including system) may have negative behavior 
>>> I think is fine
>> This reminds me of the current situation with our tests where we just keep 
>> adding more and more without really considering the value of the current set 
>> and the costs of that body of work as it keeps growing.
>> 
>> Having some kind of signal that we need to do some housekeeping with our 
>> system tables, or something in the feedback loop that helps us keep on top 
>> of this hygiene over time, seems like a clear benefit to me.
>> 
>> On Tue, Jun 13, 2023, at 1:42 PM, David Capwell wrote:
>>>> I think that the combined decision of using a default value and counting 
>

Re: [DISCUSS] Remove deprecated keyspace_count_warn_threshold and table_count_warn_threshold

[David Capwell]

> I think that the combined decision of using a default value and counting 
> system tables was a mistake

Default value I agree with you; features should be off by default!  If we 
remove the default then we disable the feature by default (which im cool with) 
and for anyone who changed the config, they would keep their behavior

As for system tables… each table adds a cost to our bookkeeping, so when we add 
new tables the cost grows and the memory per table decreases, does it not?  
Warning that too many tables (including system) may have negative behavior I 
think is fine, its only if we start to fail is when things become a problem 
(upgrading to 5.0 can’t happen due to too many tables added in the release?); 
think the feature was warn only, so that should be fine.  With that, I kinda 
don’t agree that including system tables is a mistake, as we add more we allow 
less for user tables before we start to have issues…. At the same time, if we 
have improvements in newer versions that allows higher number of tables, the 
user then has to update their configs (well, as long as we don’t make things 
worse a smaller limit than needed is fine…)

> we would need to know how many system keyspaces/tables were on the version we 
> are upgrading from

Do we?  The logic was pulling from local schema, so to keep the same behavior 
we would need to do the same; being version dependent would actually break the 
semantics as far as I can tell.

> On Jun 13, 2023, at 9:50 AM, Andrés de la Peña  wrote:
> 
> Indeed "keyspace_count_warn_threshold" and "table_count_warn_threshold" 
> include system keyspaces and tables. Also, differently to the newer 
> guardrails, they are enabled by default. 
> 
> I find that problematic because users need to know how many system 
> keyspaces/tables there are to know if they need to set the threshold value. 
> Moreover, if a new release adds some system tables, the threshold can start 
> to be triggered without changing the number of user tables. That would force 
> some users to update the threshold values during an upgrade. Even if they are 
> using the defaults. That situation would happen again in any release adding 
> new keyspaces/tables. I think adding new system tables is not that uncommon, 
> and indeed 5.0 does it.
> 
> I think that the combined decision of using a default value and counting 
> system tables was a mistake. If that's the case, I don't know for how long we 
> want to remain tied to that mistake. Especially when the old thresholds tend 
> to create upgrade issues on their own.
> 
> If we were going to use converters, we would need to know how many system 
> keyspaces/tables were on the version we are upgrading from. I don't know if 
> that information is available. Or perhaps we could assume that counting 
> system keyspaces/tables was a bug, and just translate changing the meaning to 
> not include them.
> 
> 
> 
> 
> 
> On Tue, 13 Jun 2023 at 16:51, David Capwell  <mailto:dcapw...@apple.com>> wrote:
>> > Have we been dropping support entirely for old params or using the 
>> > @Replaces annotation into perpetuity?
>> 
>> 
>> My understanding is that the goal is to keep things around in perpetuity 
>> unless it actively causes us harm… and with @Replaces, there tends to be no 
>> harm to keep around…
>> 
>> Looking at 
>> https://github.com/apache/cassandra/commit/bae92ee139b411c94228f8fd5bb8befb4183ca9f
>>  we just marked them deprecated and created a brand new config that matched 
>> the old… which I feel was a bad idea…. Renaming configs are fine with 
>> @Replaces, but asking users to migrate with the idea of breaking them in the 
>> future is bad…
>> 
>> The table_count_warn_threshold config is used at 
>> org.apache.cassandra.cql3.statements.schema.CreateTableStatement#clientWarnings
>> The tables_warn_threshold config is used at 
>> org.apache.cassandra.cql3.statements.schema.CreateTableStatement#validate
>> 
>> The only difference I see is that table_count_warn_threshold includes system 
>> tables where as tables_warn_threshold is only user tables…
>> 
>> > I would like to propose removing the non-guardrail thresholds 
>> > 'keyspace_count_warn_threshold' and 'table_count_warn_threshold' 
>> > configuration settings on the trunk branch for the next major release.
>> 
>> Deprecate in 4.1 is way too new for me to accept that, and its low effort to 
>> keep; breaking users is always a bad idea and doing it when not needed is 
>> bad…
>> 
>> Honestly, I don’t see why we couldn’t use @Replaces here to solve the 
>> semantic gap… table_count_warn_threshold includes the system tables, so we 
>> just need a Conv

[DISCUSS] Remove org.apache.cassandra.io.sstable.SSTableHeaderFix in trunk (5.0)?

[David Capwell]

org.apache.cassandra.io.sstable.SSTableHeaderFix was added due to bugs in 3.6 
causing invalidate types or incompatible types (due to toString changes) in the 
SSTableHeader… this logic runs on start and rewrites all Stats files that had a 
mismatch from the local schema; with 5.0 requiring upgrades from 4.x only, this 
logic should have already run as its a 3.x to 4.0 migration step (though users 
are able to opt out [1]) which should have already fixed the SSTables to have 
correct schema…

Why is this a problem now?  CASSANDRA-18504 is adding a lot of property/fuzz 
tests to the type system and the read/write path, which has found several bugs; 
fixing some of the bugs actually impacts SSTableHeader because it requires 
generating and working with types that are not valid, so it can fix them…   By 
removing this logic, we can push this type validation into the type system to 
avoid generating incorrect types.  

If we wish to keep this class, we need to maintain allowing invalid types to be 
created, which may cause bugs down the road.


[1] if a user opts out there are 2 real cases that are impacted: UDTs, and 
collections of collections…
* For UDTs, the frozen vs non-frozen type are not the same, so mixing these 
causes us to fail to read the data, failing the read…. I believe 
writes/compactions will not corrupt the data, but anything that touches these 
SSTables will fail due to the schema mismatch… the only way to resolve this is 
to fix the SSTables… If you disabled in 4.x, you were living with broken / 
unreadable SSTables, so by removing 5.0 would loose the ability to repair them 
(but 4.x would still be able to)
* for collections of collections, this is less of an issue.  The logic would 
detect that the collection has a non-frozen collection as the element, so would 
migrate them to frozen.  This behavior has been moved to the type system, so a 
read from SSTable of “list>” automagically becomes a 
"ListType(FrozenType(ListType(Int32Type)))”.  The SSTables are not “fixed”, but 
compaction is able to read the data correctly, and the new SSTables will have 
the correct header.  

Re: [DISCUSS] Remove deprecated keyspace_count_warn_threshold and table_count_warn_threshold

[David Capwell]

> Have we been dropping support entirely for old params or using the @Replaces 
> annotation into perpetuity?


My understanding is that the goal is to keep things around in perpetuity unless 
it actively causes us harm… and with @Replaces, there tends to be no harm to 
keep around…

Looking at 
https://github.com/apache/cassandra/commit/bae92ee139b411c94228f8fd5bb8befb4183ca9f
 we just marked them deprecated and created a brand new config that matched the 
old… which I feel was a bad idea…. Renaming configs are fine with @Replaces, 
but asking users to migrate with the idea of breaking them in the future is bad…

The table_count_warn_threshold config is used at 
org.apache.cassandra.cql3.statements.schema.CreateTableStatement#clientWarnings
The tables_warn_threshold config is used at 
org.apache.cassandra.cql3.statements.schema.CreateTableStatement#validate

The only difference I see is that table_count_warn_threshold includes system 
tables where as tables_warn_threshold is only user tables…

> I would like to propose removing the non-guardrail thresholds 
> 'keyspace_count_warn_threshold' and 'table_count_warn_threshold' 
> configuration settings on the trunk branch for the next major release.

Deprecate in 4.1 is way too new for me to accept that, and its low effort to 
keep; breaking users is always a bad idea and doing it when not needed is bad…

Honestly, I don’t see why we couldn’t use @Replaces here to solve the semantic 
gap… table_count_warn_threshold includes the system tables, so we just need a 
Converter that takes w/e the value the user put in and subtracts the system 
tables… which then gives us the user tables (matching tables_warn_threshold)

> On Jun 13, 2023, at 7:57 AM, Josh McKenzie  wrote:
> 
>> have subsequently been deprecated since 4.1-alpha in CASSANDRA-17195 when 
>> they were replaced/migrated to guardrails as part of CEP-3 (Guardrails).
> Have we been dropping support entirely for old params or using the @Replaces 
> annotation into perpetuity?
> 
> I dislike the idea of operators having to remember to update things between 
> versions and being surprised when things change roughly equally to us 
> carrying along undocumented deprecated param name mapping roughly equally. :)
> 
> On Mon, Jun 12, 2023, at 5:56 PM, Dan Jatnieks wrote:
>> Hello everyone,
>> 
>> I would like to propose removing the non-guardrail thresholds 
>> 'keyspace_count_warn_threshold' and 'table_count_warn_threshold' 
>> configuration settings on the trunk branch for the next major release.
>> 
>> These thresholds were first added with CASSANDRA-16309 in 4.0-beta4 and have 
>> subsequently been deprecated since 4.1-alpha in CASSANDRA-17195 when they 
>> were replaced/migrated to guardrails as part of CEP-3 (Guardrails).
>> 
>> I'd appreciate any thoughts about this. I will open a ticket to get started 
>> if there is support for doing this.
>> 
>> Reference:
>> https://issues.apache.org/jira/browse/CASSANDRA-16309
>> https://issues.apache.org/jira/browse/CASSANDRA-17195
>> CEP-3: Guardrails 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-3%3A+Guardrails
>> 
>> 
>> Thanks,
>> Dan Jatnieks

Re: [VOTE] CEP-8 Datastax Drivers Donation

[David Capwell]

+1

> On Jun 13, 2023, at 7:59 AM, Josh McKenzie  wrote:
> 
> +1
> 
> On Tue, Jun 13, 2023, at 10:55 AM, Jeremiah Jordan wrote:
>> +1 nb
>> 
>> On Jun 13, 2023 at 9:14:35 AM, Jeremy Hanna > > wrote:
>>> 
>>> Calling for a vote on CEP-8 [1].
>>> 
>>> To clarify the intent, as Benjamin said in the discussion thread [2], the 
>>> goal of this vote is simply to ensure that the community is in favor of the 
>>> donation. Nothing more.
>>> The plan is to introduce the drivers, one by one. Each driver donation will 
>>> need to be accepted first by the PMC members, as it is the case for any 
>>> donation. Therefore the PMC should have full control on the pace at which 
>>> new drivers are accepted.
>>> 
>>> If this vote passes, we can start this process for the Java driver under 
>>> the direction of the PMC.
>>> 
>>> Jeremy
>>> 
>>> 1. 
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
>>> 2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[David Capwell]

Hmmm, did you try without —remote?  We 100% rely on git hooks, and this is what 
we do

 $ grep -r 'git submodule' .build/
.build//sh/bump-accord.sh:  git submodule status modules/accord
.build//sh/change-submodule.sh:  git submodule set-url "${path}" "${url}"
.build//sh/change-submodule.sh:  git submodule set-branch --branch "${branch}" 
"${path}"
.build//sh/change-submodule.sh:  git submodule update --remote
.build//sh/development-switch.sh:  [ "$exists" == false ] && error "git 
submodule $a does not exist"
.build//sh/development-switch.sh:git submodule set-url "${path}" 
"../cassandra-${name}.git"
.build//sh/development-switch.sh:git submodule set-branch --branch 
"${branch}" "${path}"
.build//git/git-hooks/post-checkout/100-update-submodules.sh:  git submodule 
update --init —recursive

> so perhaps this a "first run tax" for submodule + worktree.

We never had such a thing in Accord… but we seem to be running commands 
slightly differently than you are...

> On Jun 1, 2023, at 1:06 PM, Josh McKenzie  wrote:
> 
>> Josh, do you see any reports on what isn’t working?  I think most people 
>> don’t touch 1% of what git can do… so it might be that 10% is broken but 
>> that no one in our domain actually touches that path?
> Was changing .gitmodule in harry to point to a branch and git just straight 
> up went out to lunch when I tried to "git submodule update --init --recursive 
> --remote" or any derivation thereof. Reproducing today in a worktree with 
> GIT_TRACE, and it looks like the submodule command is hanging on:
> 
>> 16:00:48.253406 git.c:460   trace: built-in: git index-pack 
>> --stdin --fix-thin '--keep=fetch-pack 32955 on Joshuas-MacBook-Pro.local' 
>> --check-self-contained-and-connected
>> 
> 
> On a whim I just let it run and it finally got unstuck after probably 5+ 
> minutes; this might just be down to me being impatient and the default 
> logging on git being... completely silent. =/
> 
> Looks like subsequent runs aren't hanging on that and are hopping right 
> through, so perhaps this a "first run tax" for submodule + worktree.
> 
> On Thu, Jun 1, 2023, at 2:05 PM, David Capwell wrote:
>> To be clear, we only use the relative syntax during development and not long 
>> lived feature branches like cep-15-accord; we use https address there.  So 
>> when you create a PR you switch to relative paths (if-and-only-if you change 
>> the submodule), then on merge you switch back to https pointing to apache.  
>> So the main issue has been when 2 authors try to work together (such as 
>> during review of a PR)
>> 
>>> On Jun 1, 2023, at 10:15 AM, David Capwell  wrote:
>>> 
>>> Most edge cases we have seen in Accord are working with feature branches 
>>> from other authors where we use relative paths to make sure the git@ vs 
>>> https:// doesn’t become a problem for CI (submodule points to https:// to 
>>> work in CI, but if you do that during feature development it gets annoying 
>>> to push to GitHub… so we do ../cassandra-accord.git so git respects w/e 
>>> protocol you are using).  In 1-2 peoples environments, when they checked 
>>> out another authors logic the C* remote was correct, but the Accord one was 
>>> still pointing to Apache (which doesn’t have the feature branch)…. This is 
>>> trivial to fix, and might be a bug with our git hooks…. But still calling 
>>> out as it has been an issue.
>>> 
>>> Josh, do you see any reports on what isn’t working?  I think most people 
>>> don’t touch 1% of what git can do… so it might be that 10% is broken but 
>>> that no one in our domain actually touches that path?
>>> 
>>>> On May 31, 2023, at 12:36 PM, Josh McKenzie  wrote:
>>>> 
>>>> Bumping into worktree + submodule pain on some harry related work; it 
>>>> looks like "git worktree" and submodules are not currently fully 
>>>> implemented:
>>>> 
>>>> https://git-scm.com/docs/git-worktree#_bugs
>>>> BUGS
>>>> Multiple checkout in general is still experimental, and the support for 
>>>> submodules is incomplete. It is NOT recommended to make multiple checkouts 
>>>> of a superproject.
>>>> 
>>>> I rely pretty heavily on worktrees and I know a lot of other folks who do 
>>>> too. This is a dealbreaker for me in terms of adding anything else as a 
>>>> submodule and I'd like to know if the accord folks have been running into 
>>>> any worktree rela

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[David Capwell]

To be clear, we only use the relative syntax during development and not long 
lived feature branches like cep-15-accord; we use https address there.  So when 
you create a PR you switch to relative paths (if-and-only-if you change the 
submodule), then on merge you switch back to https pointing to apache.  So the 
main issue has been when 2 authors try to work together (such as during review 
of a PR)

> On Jun 1, 2023, at 10:15 AM, David Capwell  wrote:
> 
> Most edge cases we have seen in Accord are working with feature branches from 
> other authors where we use relative paths to make sure the git@ vs https:// 
> doesn’t become a problem for CI (submodule points to https:// to work in CI, 
> but if you do that during feature development it gets annoying to push to 
> GitHub… so we do ../cassandra-accord.git so git respects w/e protocol you are 
> using).  In 1-2 peoples environments, when they checked out another authors 
> logic the C* remote was correct, but the Accord one was still pointing to 
> Apache (which doesn’t have the feature branch)…. This is trivial to fix, and 
> might be a bug with our git hooks…. But still calling out as it has been an 
> issue.
> 
> Josh, do you see any reports on what isn’t working?  I think most people 
> don’t touch 1% of what git can do… so it might be that 10% is broken but that 
> no one in our domain actually touches that path?
> 
>> On May 31, 2023, at 12:36 PM, Josh McKenzie  wrote:
>> 
>> Bumping into worktree + submodule pain on some harry related work; it looks 
>> like "git worktree" and submodules are not currently fully implemented:
>> 
>> https://git-scm.com/docs/git-worktree#_bugs
>> BUGS
>> Multiple checkout in general is still experimental, and the support for 
>> submodules is incomplete. It is NOT recommended to make multiple checkouts 
>> of a superproject.
>> 
>> I rely pretty heavily on worktrees and I know a lot of other folks who do 
>> too. This is a dealbreaker for me in terms of adding anything else as a 
>> submodule and I'd like to know if the accord folks have been running into 
>> any worktree related woes w/the accord integration.
>> 
>> 
>> On Sun, May 28, 2023, at 10:14 AM, Alex Petrov wrote:
>>> Regarding approachability, one of the things I thought is worth adding is a 
>>> DSL. I feel like there's enough functionality in Harry and there's enough 
>>> information for anyone who needs to write even an involved test out there, 
>>> but adoption doesn't usually start with complex use-cases, so it could be 
>>> that making it extremely simple to generate the data and validating that 
>>> written data is where it's supposed to be, should help adoption a lot. 
>>> Unfortunately, more complex use-cases such as group-by support, or SAI 
>>> testing will require a bit more knowledge and writing an involved model, so 
>>> I do not see any shortcuts we can take here.
>>> 
>>> > I do think that moving Harry in-tree would improve approachability
>>> 
>>> I think it's similar as it is with in-jvm dtest api. I feel like we wold 
>>> evolve it more actively if we didn't have to cut a release before every 
>>> commit. In other words, I think that changing Harry code and extending 
>>> functionality will be easier, which I think will eventually lead to quicker 
>>> adoption. But of course the act of moving itself does not increase 
>>> adoption, it just comes from better ergonomics.
>>> 
>>> 
>>> On Thu, May 25, 2023, at 8:03 PM, Abe Ratnofsky wrote:
>>>> I'm seeing a few distinct topics here:
>>>> 
>>>> 1. Harry's adoption and approachability
>>>> 
>>>> I agree that approachability is one of Harry's main improvement areas 
>>>> right now. If our goal is to produce a fuzz testing framework for the 
>>>> Cassandra project, then adoption by contributors and usage for new feature 
>>>> development are reasonable indicators for whether we're achieving that 
>>>> goal. If Harry is not getting adopted by contributors outside of Apple, 
>>>> and is not getting used for new feature development, then we should make 
>>>> an effort to understand why. I don't think that a several-hour seminar is 
>>>> the best point of leverage to achieve those goals.
>>>> 
>>>> Here's what I think we do need:
>>>> 
>>>> - The README should be understandable by anyone interested in writing a 
>>>> fuzz test
>>>> - Example tests should be runnable from a fresh clone of Cassandra, in an 
>>>> IDE or

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[David Capwell]

Most edge cases we have seen in Accord are working with feature branches from 
other authors where we use relative paths to make sure the git@ vs https:// 
doesn’t become a problem for CI (submodule points to https:// to work in CI, 
but if you do that during feature development it gets annoying to push to 
GitHub… so we do ../cassandra-accord.git so git respects w/e protocol you are 
using).  In 1-2 peoples environments, when they checked out another authors 
logic the C* remote was correct, but the Accord one was still pointing to 
Apache (which doesn’t have the feature branch)…. This is trivial to fix, and 
might be a bug with our git hooks…. But still calling out as it has been an 
issue.

Josh, do you see any reports on what isn’t working?  I think most people don’t 
touch 1% of what git can do… so it might be that 10% is broken but that no one 
in our domain actually touches that path?

> On May 31, 2023, at 12:36 PM, Josh McKenzie  wrote:
> 
> Bumping into worktree + submodule pain on some harry related work; it looks 
> like "git worktree" and submodules are not currently fully implemented:
> 
> https://git-scm.com/docs/git-worktree#_bugs
> BUGS
> Multiple checkout in general is still experimental, and the support for 
> submodules is incomplete. It is NOT recommended to make multiple checkouts of 
> a superproject.
> 
> I rely pretty heavily on worktrees and I know a lot of other folks who do 
> too. This is a dealbreaker for me in terms of adding anything else as a 
> submodule and I'd like to know if the accord folks have been running into any 
> worktree related woes w/the accord integration.
> 
> 
> On Sun, May 28, 2023, at 10:14 AM, Alex Petrov wrote:
>> Regarding approachability, one of the things I thought is worth adding is a 
>> DSL. I feel like there's enough functionality in Harry and there's enough 
>> information for anyone who needs to write even an involved test out there, 
>> but adoption doesn't usually start with complex use-cases, so it could be 
>> that making it extremely simple to generate the data and validating that 
>> written data is where it's supposed to be, should help adoption a lot. 
>> Unfortunately, more complex use-cases such as group-by support, or SAI 
>> testing will require a bit more knowledge and writing an involved model, so 
>> I do not see any shortcuts we can take here.
>> 
>> > I do think that moving Harry in-tree would improve approachability
>> 
>> I think it's similar as it is with in-jvm dtest api. I feel like we wold 
>> evolve it more actively if we didn't have to cut a release before every 
>> commit. In other words, I think that changing Harry code and extending 
>> functionality will be easier, which I think will eventually lead to quicker 
>> adoption. But of course the act of moving itself does not increase adoption, 
>> it just comes from better ergonomics.
>> 
>> 
>> On Thu, May 25, 2023, at 8:03 PM, Abe Ratnofsky wrote:
>>> I'm seeing a few distinct topics here:
>>> 
>>> 1. Harry's adoption and approachability
>>> 
>>> I agree that approachability is one of Harry's main improvement areas right 
>>> now. If our goal is to produce a fuzz testing framework for the Cassandra 
>>> project, then adoption by contributors and usage for new feature 
>>> development are reasonable indicators for whether we're achieving that 
>>> goal. If Harry is not getting adopted by contributors outside of Apple, and 
>>> is not getting used for new feature development, then we should make an 
>>> effort to understand why. I don't think that a several-hour seminar is the 
>>> best point of leverage to achieve those goals.
>>> 
>>> Here's what I think we do need:
>>> 
>>> - The README should be understandable by anyone interested in writing a 
>>> fuzz test
>>> - Example tests should be runnable from a fresh clone of Cassandra, in an 
>>> IDE or on the command line
>>> - Examples of how we would test new features (like CEP-7, CEP-29, etc) with 
>>> the fuzz testing framework
>>> 
>>> I find the JVM dtest framework accomplishes similar goals, and one reason 
>>> is because there are plenty of examples, and it's relatively easy to copy 
>>> and paste one example and have it do what you'd like. I believe the same 
>>> approach would work for a fuzz testing framework.
>>> 
>>> Some of these tasks above are already done for Harry, such as better IDE 
>>> support for samples. This will be available in OSS Harry shortly.
>>> 
>>> 2. Moving Harry in-tree vs. in submodule
>>> 
>>> As I understand it, making Harry a submodule of Cassandra would make it 
>>> easier to deal with versioning, since we wouldn't have to do the entire 
>>> release dance we need to do for dtest-api, but I don't see this as a big 
>>> improvement to approachability.
>>> 
>>> I do think that moving Harry in-tree would improve approachability, for the 
>>> same reason as the JVM dtests. It's nice to write a feature or fix, find a 
>>> similar JVM dtest, copy, paste, and edit, and have something 

Re: [VOTE] CEP-30 ANN Vector Search

[David Capwell]

+1

> On May 25, 2023, at 1:53 PM, Ekaterina Dimitrova  
> wrote:
> 
> +1
> 
> On Thu, 25 May 2023 at 16:46, Brandon Williams  > wrote:
>> +1
>> 
>> Kind Regards,
>> Brandon
>> 
>> On Thu, May 25, 2023 at 10:45 AM Jonathan Ellis > > wrote:
>> >
>> > Let's make this official.
>> >
>> > CEP: 
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes
>> >
>> > POC that demonstrates all the big rocks, including distributed queries: 
>> > https://github.com/datastax/cassandra/tree/cep-vsearch
>> >
>> > --
>> > Jonathan Ellis
>> > co-founder, http://www.datastax.com 
>> > @spyced

Re: Agrona vs fastutil and fastutil-concurrent-wrapper

[David Capwell]

Agrona isn’t going anywhere due to the library being more than basic 
collections.

Now, with regard to single-threaded collections… honestly I dislike Agrona as I 
always fight to avoid boxing; carrot was far better with this regard…. Didn’t 
look at the fastutil versions to see if they are better here, but I do know I 
am personally not happy with Agrona primitive collections…

I do believe the main motivator for this is that fastutil has a concurrent 
version of their collections, so you gain access to concurrent primitive 
collections; something we do not have today… Given the desire for concurrent 
primitive collections, I am cool with it.

> I’m not inclined to swap it out

When it came to random testing libraries, I believe the stance taken before was 
that we should allow multiple versions and the best one will win eventually… so 
I am cool having the same stance for primitive collections...

> On May 25, 2023, at 8:50 AM, Benedict  wrote:
> 
> Given they provide no data or explanation, and that benchmarking is hard, I’m 
> not inclined to give much weight to their analysis.
> 
> Agrona was favoured in large part due to the perceived quality of the 
> library. I’m not inclined to swap it out without proper evidence the 
> fastutils is both materially faster in a manner care about and of similar 
> quality.
> 
>> On 25 May 2023, at 16:43, Jonathan Ellis  wrote:
>> 
>> 
>> Try it out and see, the only data point I have is that the company who has 
>> spent more effort here than anyone else I could find likes fastutil better.
>> 
>> On Thu, May 25, 2023 at 10:33 AM Dinesh Joshi > > wrote:
>>> > On May 25, 2023, at 6:14 AM, Jonathan Ellis >> > > wrote:
>>> > 
>>> > Any objections to adding the concurrent wrapper and switching out agrona 
>>> > for fastutil?
>>> 
>>> How does fastutil compare to agrona in terms of memory profile and runtime 
>>> performance? How invasive would it be to switch?
>> 
>> 
>> -- 
>> Jonathan Ellis
>> co-founder, http://www.datastax.com 
>> @spyced

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[David Capwell]

> The time spent on getting that running has been a fair few hours, where we 
> could have cut many manual module releases in that time. 

We spent a few hours getting submodules working, and we no longer need to 
release for every single commit…

$ git log b9025e59395f47535e4ed1fec20b1186cdb07db8..HEAD | grep 'commit ' | wc 
-l
  12

So looking at accord trunk, we needed 12 votes for a release, and each vote is 
a min of 3 days, so 36 days of overhead vs 5 hours of work?

There are some hiccups, but this is mostly in the “never did this before, how 
do I setup” case, so something that prob can be improved… once you do your 
first patch the issues kinda go away.  

One thing that can be annoying is for people who don’t use work trees and 
switch between trunk and cassandra-4.x in the same directory… I am not sure if 
the issues here are my scripts, or git getting confused…. If you use work trees 
(I strongly recommend regardless of submodules or not) you don’t have these 
issues (my disk layout is below [1]).


> I'd like to discuss bringing cassandra-harry in-tree as a submodule

For accord, the main reason to keep it out of tree was to allow other projects 
to use the library (similar to RAFT libraries that exist for projects to use), 
but my mental model for Harry is that most of the code is Cassandra specific 
(models, converting timestamps to Cassandra data, etc.), so wondering if it 
makes sense in its own repo vs being in trunk directly?  Submodules do have 
their own overhead and edge cases, so I am mostly in favor of using for cases 
where the code must live outside of tree (such as jvm-dtest that lives out of 
tree as all branches need the same interfaces)



[1] I have a single git repo and use git worktrees to keep each branch in a 
isolated directory (this avoids the .git overhead in every directory)… my 
layout is

$ ls
3.0 3.114.0 4.1 cep-15-accord   
prs trunk
$ ls prs
prs:
4.1 cep-15-accord   trunk


> On May 24, 2023, at 7:53 AM, Josh McKenzie  wrote:
> 
>> I have submitted a proposal to Cassandra Summit for a 4-hour Harry workshop,
> I'm about to need to harry test for the paging across tombstone work for 
> https://issues.apache.org/jira/browse/CASSANDRA-18424 (that's where my own 
> overlapping fuzzing came in). In the process, I'll see if I can't distill 
> something really simple along the lines of how React approaches it 
> (https://react.dev/learn).
> 
> Ideally we'd be able to get something together that's a high level "In the 
> next 15 minutes, you will know and understand A-G and have access to N% of 
> the power of harry" kind of offer.
> 
> Honestly, there's a lot in our ecosystem where we could benefit from taking a 
> page from their book in terms of onboarding and getting started IMO.
> 
> On Wed, May 24, 2023, at 10:31 AM, Alex Petrov wrote:
>> > I wonder if a mini-onboarding session would be good as a community session 
>> > - go over Harry, how to run it, how to add a test?  Would that be the 
>> > right venue?  I just would like to see how we can not only plug it in to 
>> > regular CI but get everyone that wants to add a test be able to know how 
>> > to get started with it.
>> 
>> I have submitted a proposal to Cassandra Summit for a 4-hour Harry workshop, 
>> but unfortunately it got declined. Goes without saying, we can still do it 
>> online, time and resources permitting. But again, I do not think it should 
>> be barring us from making Harry a part of the codebase, as it already is. In 
>> fact, we can be iterating on the development quicker having it in-tree. 
>> 
>> We could go over some interesting examples such as testing 2i (SAI), 
>> modelling Group By tests, or testing repair. If there is enough appetite and 
>> collaboration in the community, I will see if we can pull something like 
>> that together. Input on _what_ you would like to see / hear / tested is also 
>> appreciated. Harry was developed out of a strong need for large-scale 
>> testing, which also has informed many of its APIs, but we can make it easier 
>> to access for interactive testing / unit tests. We have been doing a lot of 
>> that with Transactional Metadata, too. 
>> 
>> > I'll hold off on this until Alex Petrov chimes in. @Alex -> got any 
>> > thoughts here?
>> 
>> Yes, sorry for not responding on this thread earlier. I can not understate 
>> how excited I am about this, and how important I think this is. Time 
>> constraints are somehow hard to overcome, but I hope the results brought by 
>> TCM will make it all worth it.
>> 
>> On Wed, May 24, 2023, at 4:23 PM, Alex Petrov wrote:
>>> I think pulling Harry into the tree will make adoption easier for the 
>>> folks. I have been a bit swamped with Transactional Metadata work, but I 
>>> wanted to make some of the things we were using for testing TCM available 
>>> outside of TCM branch. This includes a bunch of helper methods to perform 
>>> operations on 

Re: Vector search demo, and query syntax

[David Capwell]

I am ok with the syntax, but wondering if a function maybe better than a CQL 
change?

SELECT id, start, end, text
FROM {self.keyspace}.{self.table}
ORDER BY ANN(embedding, ?)
LIMIT ?

Not really a common syntax, but could be useful down the line

> On May 23, 2023, at 12:37 AM, Mick Semb Wever  wrote:
> 
>> I propose that we adopt `ORDER BY` syntax, supporting it for vector indexes 
>> first and eventually for all SAI indexes.  So this query would become
>> 
>> SELECT id, start, end, text 
>> FROM {self.keyspace}.{self.table} 
>> ORDER BY embedding ANN OF %s 
>> LIMIT %s
> 
> 
> LGTM.
> 
> I first stumbled a bit with "there's no where clause and no filtering 
> allowed…" 
> 
> But I doubt that reaction from any experienced cql user will last more than a 
> moment.  
> 

Re: CEP-30: Approximate Nearest Neighbor(ANN) Vector Search via Storage-Attached Indexes

[David Capwell]

e the top 1 result.  How is the scoring done and what happens if there 
>>> are 20 that have the same score?  How will the coordinator decide which 1 
>>> is returned out of 20?
>>> 
>>> It returns results in token/partition and then clustering order.
>>> 
>>>> On May 9, 2023, at 2:53 PM, Caleb Rackliffe >>> <mailto:calebrackli...@gmail.com>> wrote:
>>>> 
>>>> Anyone on this ML who still remembers DSE Search (or has experience w/ 
>>>> Elastic or SolrCloud) probably also knows that there are some significant 
>>>> pieces of an optimized scatter/gather apparatus for IR (even without 
>>>> sorting, which also doesn't exist yet) that do not exist in C* or it's 
>>>> range query system (which SAI and all other 2i implementations use). SAI, 
>>>> like all C* 2i implementations, is still a local index, and as that is the 
>>>> case, anything built on it will perform best in partition-scoped (at least 
>>>> on the read side) use-cases. (On the bright side, the project is moving 
>>>> toward larger partitions being a possibility.) With smaller clusters or 
>>>> use-cases that are extremely write-heavy/read-light, it's possible that 
>>>> the full scatter/gather won't be too onerous, especially w/ a few small 
>>>> tweaks (on top of a non-vnode cluster) to a.) keep fanout minimal and b.) 
>>>> keep range/index queries to a single pass to minimize latency.
>>>> 
>>>> Whatever we do, we just need to avoid a situation down the road where 
>>>> users don't understand these nuances and hit a wall where they try to use 
>>>> this in a way that is fundamentally incompatible w/ the way the database 
>>>> scales/works. (I've done my best to call this out in all discussions 
>>>> around SAI over time, and there may even end up being further guardrails 
>>>> put in place to make it even harder to misuse it...but I digress.)
>>>> 
>>>> Having said all that, I don't fundamentally have a problem w/ the proposal.
>>>> 
>>>> On Tue, May 9, 2023 at 2:11 PM Benedict >>> <mailto:bened...@apache.org>> wrote:
>>>>> HNSW can in principle be made into a distributed index. But that would be 
>>>>> quite a different paradigm to SAI.
>>>>> 
>>>>>> On 9 May 2023, at 19:30, Patrick McFadin >>>>> <mailto:pmcfa...@gmail.com>> wrote:
>>>>>> 
>>>>>> 
>>>>>> Under the goals section, there is this line:
>>>>>> 
>>>>>> Scatter/gather across replicas, combining topK from each to get global 
>>>>>> topK.
>>>>>> 
>>>>>> But what I'm hearing is, exactly how will that happen? Maybe this is an 
>>>>>> SAI question too. How is that verified in SAI?
>>>>>> 
>>>>>> On Tue, May 9, 2023 at 11:07 AM David Capwell >>>>> <mailto:dcapw...@apple.com>> wrote:
>>>>>>> Approach section doesn’t go over how this will handle cross replica 
>>>>>>> search, this would be good to flesh out… given results have a real 
>>>>>>> ranking, the current 2i logic may yield incorrect results… so would 
>>>>>>> think we need num_ranges / rf queries in the best case, with some new 
>>>>>>> capability to sort the results?  If my assumption is correct, then how 
>>>>>>> errors are handled should also be fleshed out… Example: 1k cluster 
>>>>>>> without vnode and RF=3, so 333 queries fanned out to match, then 
>>>>>>> coordinator needs to sort… if 1 of the queries fails and can’t fall 
>>>>>>> back to peers… does the query fail (I assume so)?
>>>>>>> 
>>>>>>>> On May 8, 2023, at 7:20 PM, Jonathan Ellis >>>>>>> <mailto:jbel...@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> Hi all,
>>>>>>>> 
>>>>>>>> Following the recent discussion threads, I would like to propose 
>>>>>>>> CEP-30 to add Approximate Nearest Neighbor (ANN) Vector Search via 
>>>>>>>> Storage-Attached Indexes (SAI) to Apache Cassandra.
>>>>>>>> 
>>>>>>>> The primary goal of this proposal is to implement ANN vector search 
>>>>>>>> capabilities, making Cassandra more useful to AI developers and 
>>>>>>>> organizations managing large datasets that can benefit from fast 
>>>>>>>> similarity search.
>>>>>>>> 
>>>>>>>> The implementation will leverage Lucene's Hierarchical Navigable Small 
>>>>>>>> World (HNSW) library and introduce a new CQL data type for vector 
>>>>>>>> embeddings, a new SAI index for ANN search functionality, and a new 
>>>>>>>> CQL operator for performing ANN search queries.
>>>>>>>> 
>>>>>>>> We are targeting the 5.0 release for this feature, in conjunction with 
>>>>>>>> the release of SAI. The proposed changes will maintain compatibility 
>>>>>>>> with existing Cassandra functionality and compose well with the 
>>>>>>>> already-approved SAI features.
>>>>>>>> 
>>>>>>>> Please find the full CEP document here: 
>>>>>>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes
>>>>>>>> 
>>>>>>>> -- 
>>>>>>>> Jonathan Ellis
>>>>>>>> co-founder, http://www.datastax.com <http://www.datastax.com/>
>>>>>>>> @spyced
>>>>>>> 
>>> 
>> 

Re: [DISCUSS] The future of CREATE INDEX

[David Capwell]

> [POLL] Centralize existing syntax or create new syntax?


1.) CREATE INDEX ... USING  WITH OPTIONS...

> [POLL] Should there be a default? (YES/NO)

Yes

> [POLL] What do do with the default?

3.) YAML config to override default index (legacy 2i remains the default)
4.) YAML config/guardrail to require index type selection (not required by 
default)

For me 3 AND 4.  When no type is given allow a config for the default, and add 
a guardrail to limit what index types are allowed.. if I misunderstood 4, I 
still prefer my option that we should have a allow list of types an operator is 
willing to support

> On May 15, 2023, at 7:39 AM, Patrick McFadin  wrote:
> 
> 1
> Yes
> 4
> 
> 
> 
> On Mon, May 15, 2023 at 3:00 AM Benedict  > wrote:
>> 3: CREATE  INDEX (Otherwise 2)
>> No
>> If configurable, should be a distributed configuration. This is very 
>> different to other local configurations, as the 2i selected has semantic 
>> implications, not just performance (and the perf implications are also much 
>> greater)
>> 
>>> On 15 May 2023, at 10:45, Mike Adamson >> > wrote:
>>> 
>>> 
 [POLL] Centralize existing syntax or create new syntax?
 
 1.) CREATE INDEX ... USING  WITH OPTIONS...
 2.) CREATE LOCAL INDEX ... USING ... WITH OPTIONS...  (same as 1, but adds 
 LOCAL keyword for clarity and separation from future GLOBAL indexes)
>>>  
>>> 1.) CREATE INDEX ... USING  WITH OPTIONS...
>>> 
 [POLL] Should there be a default? (YES/NO)
>>> 
>>> Yes
>>> 
 [POLL] What do do with the default?
 
 1.) Allow a default, and switch it to SAI (no configurables)
 2.) Allow a default, and stay w/ the legacy 2i (no configurables)
 3.) YAML config to override default index (legacy 2i remains the default)
 4.) YAML config/guardrail to require index type selection (not required by 
 default)
>>> 
>>> 3.) YAML config to override default index (legacy 2i remains the default)
>>> 
>>> 
>>> 
>>> On Mon, 15 May 2023 at 08:54, Mick Semb Wever >> > wrote:
 
 
> [POLL] Centralize existing syntax or create new syntax?
> 
> 1.) CREATE INDEX ... USING  WITH OPTIONS...
> 2.) CREATE LOCAL INDEX ... USING ... WITH OPTIONS...  (same as 1, but 
> adds LOCAL keyword for clarity and separation from future GLOBAL indexes)
 
 
 (1) CREATE INDEX …
 
  
> [POLL] Should there be a default? (YES/NO)
 
 
 Yes (but see below).
 
  
> [POLL] What do do with the default?
> 
> 1.) Allow a default, and switch it to SAI (no configurables)
> 2.) Allow a default, and stay w/ the legacy 2i (no configurables)
> 3.) YAML config to override default index (legacy 2i remains the default)
> 4.) YAML config/guardrail to require index type selection (not required 
> by default)
 
 
 (4) YAML config. Commented out default of 2i.
 
 I agree that the default cannot change in 5.0, but our existing default of 
 2i can be commented out.
 
 For the user this gives them the same feedback, and puts the same 
 requirement to edit one line of yaml, as when we disabled MVs and SASI in 
 4.0
 No one has complained about either of these, which is a clear signal folk 
 understood how to get their existing DDLs to work from 3.x to 4.x
>>> 
>>> 
>>> -- 
>>>  Mike Adamson
>>> Engineering
>>> 
>>> +1 650 389 6000  | datastax.com 
>>> Find DataStax Online:
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 

Re: [DISCUSS] The future of CREATE INDEX

[David Capwell]

> I really dislike the idea of the same CQL doing different things based upon a 
> per-node configuration.

> I agree with Brandon that changing CQL behaviour like this based on node 
> config is really not ideal. 

I am cool adding such a config, and also cool keeping CREATE INDEX disabled by 
default…. But would like to point out that we have many configs that impact CQL 
and they are almost always local configs…

Is CREATE INDEX even allowed?  This is a per node config. Right now you can 
block globally, enable on a single instance, create the index for your users, 
then revert the config change on the instance…. 

All guardrails that define what we can do are per node configs…

Now, giving this thread, there is pushback for a config to allow default impl 
to change… but there is 0 pushback for new syntax to make this explicit…. So 
maybe we should [POLL] for what syntax people want?

> if we decide before the 5.0 release that we have enough information to change 
> the default (#1), we can change it in a matter of minutes.

I am strongly against this… SAI is new for 5.0 so should be disabled by 
default; else we disrespect the idea that new features are disabled by default. 
 I am cool with our docs recommending if we do find its better in most cases, 
but we should not change the default in the same reason it lands in.

> On May 12, 2023, at 10:10 AM, Caleb Rackliffe  
> wrote:
> 
> I don't want to cut over for 5.0 either way. I was more contrasting a 
> configurable cutover in 5.0 vs. a hard cutover later.
> 
> On Fri, May 12, 2023 at 12:09 PM Benedict  > wrote:
>> If the performance characteristics are as clear cut as you think, then maybe 
>> it will be an easy decision once the evidence is available for everyone to 
>> consider?
>> 
>> If not, then we probably can’t do the hard cutover and so the answer is 
>> still pretty simple? 
>> 
>>> On 12 May 2023, at 18:04, Caleb Rackliffe >> > wrote:
>>> 
>>> 
>>> I don't particularly like the YAML solution either, but absent that, we're 
>>> back to fighting about whether we introduce entirely new syntax or hard cut 
>>> over to SAI at some point.
>>> 
>>> We already have per-node configuration in the YAML that determines whether 
>>> or not we can create a 2i at all, right?
>>> 
>>> What if we just do #2 and #3 and punt on everything else?
>>> 
>>> On Fri, May 12, 2023 at 11:56 AM Benedict >> > wrote:
 A table is not a local concept at all, it has a global primary index - 
 that’s the core idea of Cassandra.
 
 I agree with Brandon that changing CQL behaviour like this based on node 
 config is really not ideal. New syntax is by far the simplest and safest 
 solution to this IMO. It doesn’t have to use the word LOCAL, but I think 
 that’s anyway an improvement, personally. 
 
 In future we will hopefully offer GLOBAL indexes, and IMO it is better to 
 reify the distinction in the syntax.
 
> On 12 May 2023, at 17:29, Caleb Rackliffe  > wrote:
> 
> 
> We don't need to know everything about SAI's performance profile to plan 
> and execute some small, reasonable things now for 5.0. I'm going to try 
> to summarize the least controversial package of ideas from the discussion 
> above. I've left out creating any new syntax. For example, I think CREATE 
> LOCAL INDEX, while explicit, is just not necessary. We don't use CREATE 
> LOCAL TABLE, although it has the same locality as our indexes.
> 
> Okay, so the proposal for 5.0...
> 
> 1.) Add a YAML option that specifies a default implementation for CREATE 
> INDEX, and make this the legacy 2i for now. No existing DDL breaks. We 
> don't have to commit to the absolute superiority of SAI.
> 2.) Add USING...WITH... support to CREATE INDEX, so we don't have to go 
> to market using CREATE CUSTOM INDEX, which feels...not so polished. (The 
> backend for this already exists w/ CREATE CUSTOM INDEX.)
> 3.) Leave in place but deprecate (client warnings could work?) CREATE 
> CUSTOM INDEX. Support the syntax for the foreseeable future.
> 
> Can we live w/ this?
> 
> I don't think any information about SAI we could possibly acquire before 
> a 5.0 release would affect the reasonableness of this much.
> 
> 
> On Fri, May 12, 2023 at 10:54 AM Benedict  > wrote:
>>> if we didn't have copious amounts of (not all public, I know, working 
>>> on it) evidence
>> 
>> If that’s the assumption on which this proposal is based, let’s discuss 
>> the evidence base first, as given the fundamentally different way they 
>> work (almost diametrically opposite), I would want to see a very high 
>> quality of evidence to support the claim.
>> 
>> I don’t think we can resolve this conversation 

Re: [VOTE] CEP-29 CQL NOT Operator

[David Capwell]

+1

> On May 10, 2023, at 9:36 AM, Francisco Guerrero  wrote:
> 
> +1 (nb)
> 
> On 2023/05/10 14:10:06 Jeremiah D Jordan wrote:
>> +1 nb
>> 
>>> On May 8, 2023, at 3:52 AM, Piotr Kołaczkowski  
>>> wrote:
>>> 
>>> Let's vote.
>>> 
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-29%3A+CQL+NOT+operator
>>> 
>>> Piotr Kołaczkowski
>>> e. pkola...@datastax.com
>>> w. www.datastax.com
>> 
>> 

Re: [DISCUSS] The future of CREATE INDEX

[David Capwell]

> Having to revert to CREATE CUSTOM INDEX sounds pretty awful, so I'd prefer 
> allowing USING...WITH... for CREATE INDEX

I have 0 issues with a new syntax to make this more clear

> just deprecating CREATE CUSTOM INDEX (at least after 5.0), but that's more or 
> less what my original proposal was above (modulo the configurable default).

I have 0 issues deprecating and producing a ClientWarning recommending the new 
syntax, but I would be against removing this syntax later on… it should be low 
effort to keep, so breaking a user would not be desirable for me.

> change only the fact that CREATE INDEX retains a configurable default


This option allows users to control this behavior, and allows us to change the 
default over time.  For 5.0 I am strongly against SAI being the default (new 
features disabled by default), but I wouldn’t have issues in later versions 
changing the default once its been out for awhile.

> I’m not convinced by the changing defaults argument here. The characteristics 
> of the two index types are very different, and users with scripts that make 
> indexes today shouldn’t have their behaviour change.

In my mind this is no different from defaulting to BTI in a follow up release, 
but if this concern is that the legacy index leaked details such as index 
tables, so changing the default would have side effects in the public domain 
that users might not expect, then I get it… are there other concerns?

> On May 10, 2023, at 9:03 AM, Caleb Rackliffe  wrote:
> 
> tl;dr If you take my original proposal and change only the fact that CREATE 
> INDEX retains a configurable default, I think we get to the same place?
> 
> (Then it's just a matter of what we do in 5.0 vs. after 5.0...)
> 
> On Wed, May 10, 2023 at 11:00 AM Caleb Rackliffe  > wrote:
>> I see a broad desire here to have a configurable (YAML) default 
>> implementation for CREATE INDEX. I'm not strongly opposed to that, as the 
>> concept of a default index implementation is pretty standard for most DBMS 
>> (see Postgres, etc.). However, keep in mind that if we do that, we still 
>> need to either revert to CREATE CUSTOM INDEX or add the USING...WITH... 
>> extensions to CREATE INDEX to override the default or specify parameters, 
>> which will be in play once SAI supports basic text tokenization/filtering. 
>> Having to revert to CREATE CUSTOM INDEX sounds pretty awful, so I'd prefer 
>> allowing USING...WITH... for CREATE INDEX and just deprecating CREATE CUSTOM 
>> INDEX (at least after 5.0), but that's more or less what my original 
>> proposal was above (modulo the configurable default).
>> 
>> Thoughts?
>> 
>> On Wed, May 10, 2023 at 2:59 AM Benedict > > wrote:
>>> I’m not convinced by the changing defaults argument here. The 
>>> characteristics of the two index types are very different, and users with 
>>> scripts that make indexes today shouldn’t have their behaviour change.
>>> 
>>> We could introduce new syntax that properly appreciates there’s no default 
>>> index, perhaps CREATE LOCAL [type] INDEX? To also make clear that these 
>>> indexes involve a partition key or scatter gather
>>> 
 On 10 May 2023, at 06:26, guo Maxwell >>> > wrote:
 
 
 +1 , as we must Improve the image of your own default indexing ability.
 
 and As for CREATE CUSTOM INDEX , should we just left as it is and we can 
 disable the ability for create SAI through  CREATE CUSTOM INDEX  in some 
 version after 5.0? 
 
 for as I know there may be users using this as a plugin-index interface, 
 like https://github.com/Stratio/cassandra-lucene-index (though these 
 project may be inactive， But if someone wants to do something similar in 
 the future, we don't have to stop).
 
 
 
 Jonathan Ellis mailto:jbel...@gmail.com>> 
 于2023年5月10日周三 10:01写道：
> +1 for this, especially in the long term.  CREATE INDEX should do the 
> right thing for most people without requiring extra ceremony.
> 
> On Tue, May 9, 2023 at 5:20 PM Jeremiah D Jordan 
> mailto:jeremiah.jor...@gmail.com>> wrote:
>> If the consensus is that SAI is the right default index, then we should 
>> just change CREATE INDEX to be SAI, and legacy 2i to be a CUSTOM INDEX.
>> 
>> 
>>> On May 9, 2023, at 4:44 PM, Caleb Rackliffe >> > wrote:
>>> 
>>> Earlier today, Mick started a thread on the future of our index 
>>> creation DDL on Slack:
>>> 
>>> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019
>>> 
>>> At the moment, there are two ways to create a secondary index.
>>> 
>>> 1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()
>>> 
>>> This creates an optionally named legacy 2i on the provided table and 
>>> column.
>>> 
>>> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
>>> 
>>> 

Re: [DISCUSS] The future of CREATE INDEX

[David Capwell]

If we assume SAI is what we should use by default for the cluster, would it 
make sense to allow

CREATE INDEX [IF NOT EXISTS] [name] ON  ()

But use a new yaml config that switches from legacy to SAI?

default_2i_impl: sai

For 5.0 we can default to “legacy” (new features disabled by default), but 
allow operators to change this to SAI if they desire?

> 2.) Leave CREATE CUSTOM INDEX...USING... available by default.

For 5.0, I would argue all indexes should be disabled by default and require 
operators to allow… I am totally cool with a new allow list to allow some impl..

secondary_indexes_enabled: false
secondary_indexes_impl_allowed: [] # default, but could allow users to do 
[’sai’] if they wish to allow sai… this does have weird semantics as it causes 
_enabled to be ignored… this could also replace _enabled, but what is allowed 
in the true case isn’t 100% clear?  Maybe you need _enabled=true and this allow 
list limits what is actually allowed (prob is way more clear)?


> 2.) Replace both CREATE INDEX and CREATE CUSTOM INDEX w/ something of a 
> hybrid between the two. For example, CREATE INDEX...USING...WITH. This would 
> both be flexible enough to accommodate index implementation selection and 
> prescriptive enough to force the user to make a decision (and wouldn't change 
> the legacy behavior of the existing CREATE INDEX). In this world, creating a 
> legacy 2i might look something like CREATE INDEX...USING `legacy`.

I do not mind a new syntax that tries to be more clear, but the “replace” is 
what I would push back against… we should keep the 2 existing syntax and not 
force users to migrate… we can logically merge the 3 syntaxes, but we should 
not remove the 2 others.

CREATE INDEX - gets rewritten to CREATE INDEX… USING config.default_2i_imp
CREATE CUSTOM INDEX` - gets rewritten to new using syntax

> 3.) Eventually deprecate CREATE CUSTOM INDEX…USING.

I don’t mind producing a warning telling users its best to use the new syntax, 
but if its low effort for us to maintain, we should… and since this can be 
rewritten to the new format in the parser, this should be low effort to 
support, so we should?

> On May 9, 2023, at 2:44 PM, Caleb Rackliffe  wrote:
> 
> Earlier today, Mick started a thread on the future of our index creation DDL 
> on Slack:
> 
> https://the-asf.slack.com/archives/C018YGVCHMZ/p1683527794220019
> 
> At the moment, there are two ways to create a secondary index.
> 
> 1.) CREATE INDEX [IF NOT EXISTS] [name] ON  ()
> 
> This creates an optionally named legacy 2i on the provided table and column.
> 
> ex. CREATE INDEX my_index ON kd.tbl(my_text_col)
> 
> 2.) CREATE CUSTOM INDEX [IF NOT EXISTS] [name] ON  () USING 
>  [WITH OPTIONS = ]
> 
> This creates a secondary index on the provided table and column using the 
> specified 2i implementation class and (optional) parameters.
> 
> ex. CREATE CUSTOM INDEX my_index ON ks.tbl(my_text_col) USING 
> 'StorageAttachedIndex'
> 
> (Note that the work on SAI added aliasing, so `StorageAttachedIndex` is 
> shorthand for the fully-qualified class name, which is also valid.)
> 
> So what is there to discuss?
> 
> The concern Mick raised is...
> 
> "...just folk continuing to use CREATE INDEX  because they think CREATE 
> CUSTOM INDEX is advanced (or just don't know of it), and we leave users doing 
> 2i (when they think they are, and/or we definitely want them to be, using 
> SAI)"
> 
> To paraphrase, we want people to use SAI once it's available where possible, 
> and the default behavior of CREATE INDEX could be at odds w/ that.
> 
> The proposal we seem to have landed on is something like the following:
> 
> For 5.0:
> 
> 1.) Disable by default the creation of new legacy 2i via CREATE INDEX.
> 2.) Leave CREATE CUSTOM INDEX...USING... available by default.
> 
> (Note: How this would interact w/ the existing secondary_indexes_enabled YAML 
> options isn't clear yet.)
> 
> Post-5.0:
> 
> 1.) Deprecate and eventually remove SASI when SAI hits full feature parity w/ 
> it.
> 2.) Replace both CREATE INDEX and CREATE CUSTOM INDEX w/ something of a 
> hybrid between the two. For example, CREATE INDEX...USING...WITH. This would 
> both be flexible enough to accommodate index implementation selection and 
> prescriptive enough to force the user to make a decision (and wouldn't change 
> the legacy behavior of the existing CREATE INDEX). In this world, creating a 
> legacy 2i might look something like CREATE INDEX...USING `legacy`.
> 3.) Eventually deprecate CREATE CUSTOM INDEX...USING.
> 
> Eventually we would have a single enabled DDL statement for index creation 
> that would be minimal but also explicit/able to handle some evolution.
> 
> What does everyone think?

Re: CEP-30: Approximate Nearest Neighbor(ANN) Vector Search via Storage-Attached Indexes

[David Capwell]

Approach section doesn’t go over how this will handle cross replica search, 
this would be good to flesh out… given results have a real ranking, the current 
2i logic may yield incorrect results… so would think we need num_ranges / rf 
queries in the best case, with some new capability to sort the results?  If my 
assumption is correct, then how errors are handled should also be fleshed out… 
Example: 1k cluster without vnode and RF=3, so 333 queries fanned out to match, 
then coordinator needs to sort… if 1 of the queries fails and can’t fall back 
to peers… does the query fail (I assume so)?

> On May 8, 2023, at 7:20 PM, Jonathan Ellis  wrote:
> 
> Hi all,
> 
> Following the recent discussion threads, I would like to propose CEP-30 to 
> add Approximate Nearest Neighbor (ANN) Vector Search via Storage-Attached 
> Indexes (SAI) to Apache Cassandra.
> 
> The primary goal of this proposal is to implement ANN vector search 
> capabilities, making Cassandra more useful to AI developers and organizations 
> managing large datasets that can benefit from fast similarity search.
> 
> The implementation will leverage Lucene's Hierarchical Navigable Small World 
> (HNSW) library and introduce a new CQL data type for vector embeddings, a new 
> SAI index for ANN search functionality, and a new CQL operator for performing 
> ANN search queries.
> 
> We are targeting the 5.0 release for this feature, in conjunction with the 
> release of SAI. The proposed changes will maintain compatibility with 
> existing Cassandra functionality and compose well with the already-approved 
> SAI features.
> 
> Please find the full CEP document here: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-30%3A+Approximate+Nearest+Neighbor%28ANN%29+Vector+Search+via+Storage-Attached+Indexes
> 
> -- 
> Jonathan Ellis
> co-founder, http://www.datastax.com 
> @spyced

Re: [POLL] Vector type for ML

[David Capwell]

https://issues.apache.org/jira/browse/CASSANDRA-18504

> On May 5, 2023, at 12:27 PM, David Capwell  wrote:
> 
> Yep, fair point…. SPARSE VECTOR better maps to NON NULL MAP
> 
>> On May 5, 2023, at 11:58 AM, David Capwell  wrote:
>> 
>>> If we ever add sparse vectors, we can assume that DENSE is the default and 
>>> allow to use either DENSE, SPARSE or nothing.
>> 
>> I have been feeling that sparse is just a fixed size list with nulls… so 
>> array… if you insert {0: 42, 3: 17} then you get a array of 
>> [42, null, null, 17]?  One negative doing this is any operator/function that 
>> needs to reify large vectors (lets say 10k elements) you have a ton of 
>> memory due to us making it a array… so a new type could be used to lower 
>> this cost…
>> 
>> With DENSE VECTOR we have the syntax in place that we “could” add SPARSE 
>> later… With VECTOR we will have complications adding a sparse vector after 
>> the fact due to this implying DENSE…
>> 
>> Updated ranking
>> 
>> Syntax
>> Score
>> VECTOR
>> 21
>> DENSE VECTOR
>> 12
>> type[dimension]
>> 10
>> NON NULL [dimention]
>> 8
>> VECTOR type[n]
>> 5
>> DENSE_VECTOR
>> 4
>> NON-NULL FROZEN
>> 3
>> ARRAY
>> 1
>> 
>> Syntax
>> Round 1
>> Round 2
>> VECTOR
>> 4
>> 4
>> DENSE VECTOR
>> 2
>> 3
>> NON NULL [dimention]
>> 2
>> 1
>> VECTOR type[n]
>> 1
>> 
>> type[dimension]
>> 1
>> 
>> DENSE_VECTOR
>> 1
>> 
>> NON-NULL FROZEN
>> 1
>> 
>> ARRAY
>> 0
>> 
>> 
>> VECTOR is still in the lead…
>> 
>>> On May 5, 2023, at 11:40 AM, Andrés de la Peña  wrote:
>>> 
>>> My vote is:
>>> 
>>> 1. VECTOR
>>> 2. DENSE VECTOR
>>> 3. type[dimension]
>>> 
>>> If we ever add sparse vectors, we can assume that DENSE is the default and 
>>> allow to use either DENSE, SPARSE or nothing.
>>> 
>>> Perhaps the dimension could be separated from the type, such as in 
>>> VECTOR[dimension] or VECTOR(dimension).
>>> 
>>> On Fri, 5 May 2023 at 19:05, David Capwell >> <mailto:dcapw...@apple.com>> wrote:
>>>>>> ...where, just to be clear, VECTOR means a frozen fixed 
>>>>>> size array w/ no null values?
>>>>> Assuming this is the case
>>>> 
>>>> The current agreed requirements are:
>>>> 
>>>> 1) non-null elements
>>>> 2) fixed length
>>>> 3) frozen 
>>>> 
>>>> You pointed out 3 isn’t actually required, but that would be a different 
>>>> conversation to remove =)… maybe defer this to JIRA as long as all parties 
>>>> agree in the ticket?
>>>> 
>>>> With all votes in, this is what I see
>>>> 
>>>> Syntax
>>>> Jonathan Ellis
>>>> David Capwell
>>>> Josh McKenzie
>>>> Caleb Rackliffe
>>>> Patrick McFadin
>>>> Brandon Williams
>>>> Mike Adamson
>>>> Benedict
>>>> Mick Semb Wever
>>>> Derek Chen-Becker
>>>> VECTOR
>>>> 1
>>>> 2
>>>> 2
>>>> 
>>>> 2
>>>> 1
>>>> 1
>>>> 3
>>>> 2
>>>> 
>>>> DENSE VECTOR
>>>> 2
>>>> 1
>>>> 
>>>> 
>>>> 1
>>>> 
>>>> 2
>>>> 
>>>> 
>>>> 
>>>> type[dimension]
>>>> 3
>>>> 3
>>>> 3
>>>> 1
>>>> 
>>>> 3
>>>> 
>>>> 2
>>>> 
>>>> 
>>>> DENSE_VECTOR
>>>> 
>>>> 
>>>> 1
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 3
>>>> NON NULL [dimention]
>>>> 
>>>> 1
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 1
>>>> 
>>>> 2
>>>> VECTOR type[n]
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 2
>>>> 
>>>> 
>>>> 1
>>>> 
>>>> ARRAY
>>>> 
>>>> 
>>>> 
>>>> 
>>>> 3
>>>> 
>&

Re: [POLL] Vector type for ML

[David Capwell]

Yep, fair point…. SPARSE VECTOR better maps to NON NULL MAP

> On May 5, 2023, at 11:58 AM, David Capwell  wrote:
> 
>> If we ever add sparse vectors, we can assume that DENSE is the default and 
>> allow to use either DENSE, SPARSE or nothing.
> 
> I have been feeling that sparse is just a fixed size list with nulls… so 
> array… if you insert {0: 42, 3: 17} then you get a array of 
> [42, null, null, 17]?  One negative doing this is any operator/function that 
> needs to reify large vectors (lets say 10k elements) you have a ton of memory 
> due to us making it a array… so a new type could be used to lower this cost…
> 
> With DENSE VECTOR we have the syntax in place that we “could” add SPARSE 
> later… With VECTOR we will have complications adding a sparse vector after 
> the fact due to this implying DENSE…
> 
> Updated ranking
> 
> Syntax
> Score
> VECTOR
> 21
> DENSE VECTOR
> 12
> type[dimension]
> 10
> NON NULL [dimention]
> 8
> VECTOR type[n]
> 5
> DENSE_VECTOR
> 4
> NON-NULL FROZEN
> 3
> ARRAY
> 1
> 
> Syntax
> Round 1
> Round 2
> VECTOR
> 4
> 4
> DENSE VECTOR
> 2
> 3
> NON NULL [dimention]
> 2
> 1
> VECTOR type[n]
> 1
> 
> type[dimension]
> 1
> 
> DENSE_VECTOR
> 1
> 
> NON-NULL FROZEN
> 1
> 
> ARRAY
> 0
> 
> 
> VECTOR is still in the lead…
> 
>> On May 5, 2023, at 11:40 AM, Andrés de la Peña  wrote:
>> 
>> My vote is:
>> 
>> 1. VECTOR
>> 2. DENSE VECTOR
>> 3. type[dimension]
>> 
>> If we ever add sparse vectors, we can assume that DENSE is the default and 
>> allow to use either DENSE, SPARSE or nothing.
>> 
>> Perhaps the dimension could be separated from the type, such as in 
>> VECTOR[dimension] or VECTOR(dimension).
>> 
>> On Fri, 5 May 2023 at 19:05, David Capwell > <mailto:dcapw...@apple.com>> wrote:
>>>>> ...where, just to be clear, VECTOR means a frozen fixed 
>>>>> size array w/ no null values?
>>>> Assuming this is the case
>>> 
>>> The current agreed requirements are:
>>> 
>>> 1) non-null elements
>>> 2) fixed length
>>> 3) frozen 
>>> 
>>> You pointed out 3 isn’t actually required, but that would be a different 
>>> conversation to remove =)… maybe defer this to JIRA as long as all parties 
>>> agree in the ticket?
>>> 
>>> With all votes in, this is what I see
>>> 
>>> Syntax
>>> Jonathan Ellis
>>> David Capwell
>>> Josh McKenzie
>>> Caleb Rackliffe
>>> Patrick McFadin
>>> Brandon Williams
>>> Mike Adamson
>>> Benedict
>>> Mick Semb Wever
>>> Derek Chen-Becker
>>> VECTOR
>>> 1
>>> 2
>>> 2
>>> 
>>> 2
>>> 1
>>> 1
>>> 3
>>> 2
>>> 
>>> DENSE VECTOR
>>> 2
>>> 1
>>> 
>>> 
>>> 1
>>> 
>>> 2
>>> 
>>> 
>>> 
>>> type[dimension]
>>> 3
>>> 3
>>> 3
>>> 1
>>> 
>>> 3
>>> 
>>> 2
>>> 
>>> 
>>> DENSE_VECTOR
>>> 
>>> 
>>> 1
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 3
>>> NON NULL [dimention]
>>> 
>>> 1
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 1
>>> 
>>> 2
>>> VECTOR type[n]
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 2
>>> 
>>> 
>>> 1
>>> 
>>> ARRAY
>>> 
>>> 
>>> 
>>> 
>>> 3
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NON-NULL FROZEN
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 1
>>> 
>>> Rank
>>> Weight
>>> 1
>>> 3
>>> 2
>>> 2
>>> 3
>>> 1
>>> ?
>>> 3
>>> 
>>> Syntax
>>> Score
>>> VECTOR
>>> 18
>>> DENSE VECTOR
>>> 10
>>> type[dimension]
>>> 9
>>> NON NULL [dimention]
>>> 8
>>> VECTOR type[n]
>>> 5
>>> DENSE_VECTOR
>>> 4
>>> NON-NULL FROZEN
>>> 3
>>> ARRAY
>>> 1
>>> 
>>> 
>>> Syntax
>>> Round 1
>>&g

Re: [POLL] Vector type for ML

[David Capwell]

> If we ever add sparse vectors, we can assume that DENSE is the default and 
> allow to use either DENSE, SPARSE or nothing.

I have been feeling that sparse is just a fixed size list with nulls… so 
array… if you insert {0: 42, 3: 17} then you get a array of 
[42, null, null, 17]?  One negative doing this is any operator/function that 
needs to reify large vectors (lets say 10k elements) you have a ton of memory 
due to us making it a array… so a new type could be used to lower this cost…

With DENSE VECTOR we have the syntax in place that we “could” add SPARSE later… 
With VECTOR we will have complications adding a sparse vector after the fact 
due to this implying DENSE…

Updated ranking

Syntax
Score
VECTOR
21
DENSE VECTOR
12
type[dimension]
10
NON NULL [dimention]
8
VECTOR type[n]
5
DENSE_VECTOR
4
NON-NULL FROZEN
3
ARRAY
1

Syntax
Round 1
Round 2
VECTOR
4
4
DENSE VECTOR
2
3
NON NULL [dimention]
2
1
VECTOR type[n]
1

type[dimension]
1

DENSE_VECTOR
1

NON-NULL FROZEN
1

ARRAY
0


VECTOR is still in the lead…

> On May 5, 2023, at 11:40 AM, Andrés de la Peña  wrote:
> 
> My vote is:
> 
> 1. VECTOR
> 2. DENSE VECTOR
> 3. type[dimension]
> 
> If we ever add sparse vectors, we can assume that DENSE is the default and 
> allow to use either DENSE, SPARSE or nothing.
> 
> Perhaps the dimension could be separated from the type, such as in 
> VECTOR[dimension] or VECTOR(dimension).
> 
> On Fri, 5 May 2023 at 19:05, David Capwell  <mailto:dcapw...@apple.com>> wrote:
>>>> ...where, just to be clear, VECTOR means a frozen fixed 
>>>> size array w/ no null values?
>>> Assuming this is the case
>> 
>> The current agreed requirements are:
>> 
>> 1) non-null elements
>> 2) fixed length
>> 3) frozen 
>> 
>> You pointed out 3 isn’t actually required, but that would be a different 
>> conversation to remove =)… maybe defer this to JIRA as long as all parties 
>> agree in the ticket?
>> 
>> With all votes in, this is what I see
>> 
>> Syntax
>> Jonathan Ellis
>> David Capwell
>> Josh McKenzie
>> Caleb Rackliffe
>> Patrick McFadin
>> Brandon Williams
>> Mike Adamson
>> Benedict
>> Mick Semb Wever
>> Derek Chen-Becker
>> VECTOR
>> 1
>> 2
>> 2
>> 
>> 2
>> 1
>> 1
>> 3
>> 2
>> 
>> DENSE VECTOR
>> 2
>> 1
>> 
>> 
>> 1
>> 
>> 2
>> 
>> 
>> 
>> type[dimension]
>> 3
>> 3
>> 3
>> 1
>> 
>> 3
>> 
>> 2
>> 
>> 
>> DENSE_VECTOR
>> 
>> 
>> 1
>> 
>> 
>> 
>> 
>> 
>> 
>> 3
>> NON NULL [dimention]
>> 
>> 1
>> 
>> 
>> 
>> 
>> 
>> 1
>> 
>> 2
>> VECTOR type[n]
>> 
>> 
>> 
>> 
>> 
>> 2
>> 
>> 
>> 1
>> 
>> ARRAY
>> 
>> 
>> 
>> 
>> 3
>> 
>> 
>> 
>> 
>> 
>> NON-NULL FROZEN
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 1
>> 
>> Rank
>> Weight
>> 1
>> 3
>> 2
>> 2
>> 3
>> 1
>> ?
>> 3
>> 
>> Syntax
>> Score
>> VECTOR
>> 18
>> DENSE VECTOR
>> 10
>> type[dimension]
>> 9
>> NON NULL [dimention]
>> 8
>> VECTOR type[n]
>> 5
>> DENSE_VECTOR
>> 4
>> NON-NULL FROZEN
>> 3
>> ARRAY
>> 1
>> 
>> 
>> Syntax
>> Round 1
>> Round 2
>> VECTOR
>> 3
>> 4
>> DENSE VECTOR
>> 2
>> 2
>> NON NULL [dimention]
>> 2
>> 1
>> VECTOR type[n]
>> 1
>> 
>> type[dimension]
>> 1
>> 
>> DENSE_VECTOR
>> 1
>> 
>> NON-NULL FROZEN
>> 1
>> 
>> ARRAY
>> 0
>> 
>> 
>> Under 2 different voting systems vector is in the lead and 
>> by a good amount… I have updated the patch locally to reflect this change as 
>> well.
>> 
>>> On May 5, 2023, at 10:41 AM, Mike Adamson >> <mailto:madam...@datastax.com>> wrote:
>>> 
>>>> ...where, just to be clear, VECTOR means a frozen fixed 
>>>> size array w/ no null values?
>>> Assuming this is the case, my vote is:
>>> 
>>> 1. VECTOR
>>> 2. DENSE VECTOR
>>> 
>>> I don't really have a 3rd vote because I think that type[dimension] is too 
>>> ambiguous. 
>>> 
>>> 
>>> On Fri, 5 May 2023 at 18:32, Der

Re: [POLL] Vector type for ML

[David Capwell]

>> ...where, just to be clear, VECTOR means a frozen fixed 
>> size array w/ no null values?
> Assuming this is the case

The current agreed requirements are:

1) non-null elements
2) fixed length
3) frozen 

You pointed out 3 isn’t actually required, but that would be a different 
conversation to remove =)… maybe defer this to JIRA as long as all parties 
agree in the ticket?

With all votes in, this is what I see

Syntax
Jonathan Ellis
David Capwell
Josh McKenzie
Caleb Rackliffe
Patrick McFadin
Brandon Williams
Mike Adamson
Benedict
Mick Semb Wever
Derek Chen-Becker
VECTOR
1
2
2

2
1
1
3
2

DENSE VECTOR
2
1


1

2



type[dimension]
3
3
3
1

3

2


DENSE_VECTOR


1






3
NON NULL [dimention]

1





1

2
VECTOR type[n]





2


1

ARRAY




3





NON-NULL FROZEN









1

Rank
Weight
1
3
2
2
3
1
?
3

Syntax
Score
VECTOR
18
DENSE VECTOR
10
type[dimension]
9
NON NULL [dimention]
8
VECTOR type[n]
5
DENSE_VECTOR
4
NON-NULL FROZEN
3
ARRAY
1


Syntax
Round 1
Round 2
VECTOR
3
4
DENSE VECTOR
2
2
NON NULL [dimention]
2
1
VECTOR type[n]
1

type[dimension]
1

DENSE_VECTOR
1

NON-NULL FROZEN
1

ARRAY
0


Under 2 different voting systems vector is in the lead and by 
a good amount… I have updated the patch locally to reflect this change as well.

> On May 5, 2023, at 10:41 AM, Mike Adamson  wrote:
> 
>> ...where, just to be clear, VECTOR means a frozen fixed 
>> size array w/ no null values?
> Assuming this is the case, my vote is:
> 
> 1. VECTOR
> 2. DENSE VECTOR
> 
> I don't really have a 3rd vote because I think that type[dimension] is too 
> ambiguous. 
> 
> 
> On Fri, 5 May 2023 at 18:32, Derek Chen-Becker  <mailto:de...@chen-becker.org>> wrote:
>> LOL, I'm holding you to that at the summit :) In all seriousness, I'm glad 
>> to see a robust debate around it. I guess for completeness, my order of 
>> preference is 
>> 
>> 1 - NONNULL FROZEN>
>> 2 - NONNULL TYPE (which part of this implies frozen? The NONNULL or the 
>> cardinality?)
>> 3 - DENSE_VECTOR
>> 
>> I guess my main concern with just "VECTOR" is that it's such an overloaded 
>> term. Maybe in ML it means something specific, but for anyone coming from 
>> C++, Rust, Java, etc, a Vector is both mutable and can carry null (or 
>> equivalent, e.g. None, in Rust). If the argument hadn't also been made that 
>> we should be working toward something that's not ML-specific maybe I would 
>> be less concerned.
>> 
>> Cheers,
>> 
>> Derek
>> 
>> 
>> Cheers,
>> 
>> Derek
>> 
>> On Fri, May 5, 2023 at 11:14 AM Patrick McFadin > <mailto:pmcfa...@gmail.com>> wrote:
>>> Derek, despite your preference, I would hang out with you at a party. 
>>> 
>>> On Fri, May 5, 2023 at 9:44 AM Derek Chen-Becker >> <mailto:de...@chen-becker.org>> wrote:
>>>> Speaking as someone who likes Erlang, maybe that's why I also like NONNULL 
>>>> FROZEN>. It's unambiguous what Cassandra is going to do with 
>>>> that type. DENSE VECTOR means I need to go read docs (and then probably 
>>>> double-check in the source to be sure) to be sure what exactly is going 
>>>> on. 
>>>> 
>>>> Cheers,
>>>> 
>>>> Derek
>>>> 
>>>> On Fri, May 5, 2023 at 9:54 AM Patrick McFadin >>> <mailto:pmcfa...@gmail.com>> wrote:
>>>>> I hope we are willing to consider developers that use our system because 
>>>>> if I had to teach people to use "NON-NULL FROZEN" I'm pretty 
>>>>> sure the response would be:
>>>>> 
>>>>> Did you tell me to go write a distributed map-reduce job in Erlang? I 
>>>>> beleive I did, Bob.  
>>>>> 
>>>>> On Fri, May 5, 2023 at 8:05 AM Josh McKenzie >>>> <mailto:jmcken...@apache.org>> wrote:
>>>>>> Idiomatically, to my mind, there's a question of "what space are we 
>>>>>> thinking about this datatype in"?
>>>>>> 
>>>>>> - In the context of mathematics, nullability in a vector would be 0
>>>>>> - In the context of Cassandra, nullability tends to mean a tombstone (or 
>>>>>> nothing)
>>>>>> - In the context of programming languages, it's all over the place
>>>>>> 
>>>>>> Given many models are exploring quantizing to int8 and other data types, 
>>>>>> there's definitely the "support other data types easily in the future" 
>>>>>> piece to me we need to keep in mind.
>>>>>&

Re: [POLL] Vector type for ML

[David Capwell]

Sorry, DENSE_VECTOR was pointing to the wrong row, updated score

Syntax
Score
VECTOR
16
DENSE VECTOR
11
type[dimension]
9
NON NULL [dimention]
6
VECTOR type[n]
5
DENSE_VECTOR
3
NON-NULL FROZEN
3
ARRAY
0

> On May 5, 2023, at 10:01 AM, David Capwell  wrote:
> 
> Updated
> 
> Syntax
> Jonathan Ellis
> David Capwell
> Josh McKenzie
> Caleb Rackliffe
> Patrick McFadin
> Brandon Williams
> Mike Adamson
> Benedict
> Mick Semb Wever
> Derek Chen-Becker
> VECTOR
> 1
> 2
> 2
> 
> 
> 1
> ?
> 3
> 2
> 
> DENSE VECTOR
> 2
> 1
> 
> 
> ?
> 
> ?
> 
> 
> 
> type[dimension]
> 3
> 3
> 3
> 1
> 
> 3
> 
> 2
> 
> 
> DENSE_VECTOR
> 
> 
> 1
> 
> 
> 
> 
> 
> 
> 
> NON NULL [dimention]
> 
> 1
> 
> 
> 
> 
> 
> 1
> 
> 
> VECTOR type[n]
> 
> 
> 
> 
> 
> 2
> 
> 
> 1
> 
> ARRAY
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> NON-NULL FROZEN
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 1
> 
> Rank
> Weight
> 1
> 3
> 2
> 2
> 3
> 1
> ?
> 3
> 
> Syntax
> Score
> VECTOR
> 16
> DENSE VECTOR
> 11
> type[dimension]
> 9
> DENSE_VECTOR
> 6
> NON NULL [dimention]
> 6
> VECTOR type[n]
> 5
> NON-NULL FROZEN
> 3
> ARRAY
> 0
> 
> 
> ATM VECTOR is winning with DENSE VECTOR a 
> close second.. Patrick and Mike are the swing votes… Election Day is so 
> exciting! 
> 
>> On May 5, 2023, at 9:53 AM, Mick Semb Wever  wrote:
>> 
>> 
>> 
>> On Fri, 5 May 2023 at 18:43, David Capwell > <mailto:dcapw...@apple.com>> wrote:
>>> Went through and created a spreed sheet of current votes… For Patric and 
>>> Mike, I don’t see a clear vote, so I put a ? where I “think” your 
>>> preference is… for Mick, I only put one vote as the list looked like a 
>>> summary, but you mentioned the first was your preference
>>> 
>>> Syntax
>>> Jonathan Ellis
>>> David Capwell
>>> Josh McKenzie
>>> Caleb Rackliffe
>>> Patrick McFadin
>>> Brandon Williams
>>> Mike Adamson
>>> Benedict
>>> Mick Semb Wever
>>> VECTOR
>>> 1
>>> 2
>>> 2
>>> 
>>> 
>>> 1
>>> ?
>>> 3
>>> 
>>> DENSE VECTOR
>>> 2
>>> 1
>>> 
>>> 
>>> ?
>>> 
>>> ?
>>> 
>>> 
>>> type[dimension]
>>> 3
>>> 3
>>> 3
>>> 1
>>> 
>>> 3
>>> 
>>> 2
>>> 
>>> DENSE_VECTOR
>>> 
>>> 
>>> 1
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NON NULL [dimention]
>>> 
>>> 1
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 1
>>> 
>>> VECTOR type[n]
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 2
>>> 
>>> 
>>> 1
>>> ARRAY
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> NON-NULL FROZEN
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 1 = top pick
>>> 2 = second pick
>>> 3 = third pick
>> 
>> 
>> Is what Josh writes always separate ?? 
>> 
>> My 2 is VECTOR
>> 
>> Thanks David for tallying.
>> 
> 

Re: [POLL] Vector type for ML

[David Capwell]

Updated

Syntax
Jonathan Ellis
David Capwell
Josh McKenzie
Caleb Rackliffe
Patrick McFadin
Brandon Williams
Mike Adamson
Benedict
Mick Semb Wever
Derek Chen-Becker
VECTOR
1
2
2


1
?
3
2

DENSE VECTOR
2
1


?

?



type[dimension]
3
3
3
1

3

2


DENSE_VECTOR


1







NON NULL [dimention]

1





1


VECTOR type[n]





2


1

ARRAY










NON-NULL FROZEN









1

Rank
Weight
1
3
2
2
3
1
?
3

Syntax
Score
VECTOR
16
DENSE VECTOR
11
type[dimension]
9
DENSE_VECTOR
6
NON NULL [dimention]
6
VECTOR type[n]
5
NON-NULL FROZEN
3
ARRAY
0


ATM VECTOR is winning with DENSE VECTOR a 
close second.. Patrick and Mike are the swing votes… Election Day is so 
exciting! 

> On May 5, 2023, at 9:53 AM, Mick Semb Wever  wrote:
> 
> 
> 
> On Fri, 5 May 2023 at 18:43, David Capwell  <mailto:dcapw...@apple.com>> wrote:
>> Went through and created a spreed sheet of current votes… For Patric and 
>> Mike, I don’t see a clear vote, so I put a ? where I “think” your preference 
>> is… for Mick, I only put one vote as the list looked like a summary, but you 
>> mentioned the first was your preference
>> 
>> Syntax
>> Jonathan Ellis
>> David Capwell
>> Josh McKenzie
>> Caleb Rackliffe
>> Patrick McFadin
>> Brandon Williams
>> Mike Adamson
>> Benedict
>> Mick Semb Wever
>> VECTOR
>> 1
>> 2
>> 2
>> 
>> 
>> 1
>> ?
>> 3
>> 
>> DENSE VECTOR
>> 2
>> 1
>> 
>> 
>> ?
>> 
>> ?
>> 
>> 
>> type[dimension]
>> 3
>> 3
>> 3
>> 1
>> 
>> 3
>> 
>> 2
>> 
>> DENSE_VECTOR
>> 
>> 
>> 1
>> 
>> 
>> 
>> 
>> 
>> 
>> NON NULL [dimention]
>> 
>> 1
>> 
>> 
>> 
>> 
>> 
>> 1
>> 
>> VECTOR type[n]
>> 
>> 
>> 
>> 
>> 
>> 2
>> 
>> 
>> 1
>> ARRAY
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> NON-NULL FROZEN
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 1 = top pick
>> 2 = second pick
>> 3 = third pick
> 
> 
> Is what Josh writes always separate ?? 
> 
> My 2 is VECTOR
> 
> Thanks David for tallying.
> 

Re: [POLL] Vector type for ML

[David Capwell]

Went through and created a spreed sheet of current votes… For Patric and Mike, 
I don’t see a clear vote, so I put a ? where I “think” your preference is… for 
Mick, I only put one vote as the list looked like a summary, but you mentioned 
the first was your preference

Syntax
Jonathan Ellis
David Capwell
Josh McKenzie
Caleb Rackliffe
Patrick McFadin
Brandon Williams
Mike Adamson
Benedict
Mick Semb Wever
VECTOR
1
2
2


1
?
3

DENSE VECTOR
2
1


?

?


type[dimension]
3
3
3
1

3

2

DENSE_VECTOR


1






NON NULL [dimention]

1





1

VECTOR type[n]





2


1
ARRAY









NON-NULL FROZEN










1 = top pick
2 = second pick
3 = third pick

Let me know if I am missing anyone, or if I have bad data

> On May 5, 2023, at 9:23 AM, Jonathan Ellis  wrote:
> 
> +10 for not inflicting unwieldy keywords on ML users.
> 
> Re Josh's summary, mostly agreed, my only objection to adding the DENSE 
> keyword is that I don't see a foreseeable future where we also support sparse 
> vectors, so it would end up being unnecessary extra verbosity.  So my 
> preference would be
> 
> 1. VECTOR
> 2. DENSE VECTOR (space instead of underscore, SQL isn't 
> afraid of spaces)
> 3. type[dimension]
> 
> On Fri, May 5, 2023 at 10:54 AM Patrick McFadin  <mailto:pmcfa...@gmail.com>> wrote:
>> I hope we are willing to consider developers that use our system because if 
>> I had to teach people to use "NON-NULL FROZEN" I'm pretty sure the 
>> response would be:
>> 
>> Did you tell me to go write a distributed map-reduce job in Erlang? I 
>> beleive I did, Bob.  
>> 
>> On Fri, May 5, 2023 at 8:05 AM Josh McKenzie > <mailto:jmcken...@apache.org>> wrote:
>>> Idiomatically, to my mind, there's a question of "what space are we 
>>> thinking about this datatype in"?
>>> 
>>> - In the context of mathematics, nullability in a vector would be 0
>>> - In the context of Cassandra, nullability tends to mean a tombstone (or 
>>> nothing)
>>> - In the context of programming languages, it's all over the place
>>> 
>>> Given many models are exploring quantizing to int8 and other data types, 
>>> there's definitely the "support other data types easily in the future" 
>>> piece to me we need to keep in mind.
>>> 
>>> So with the above and the "meet the user where they are and don't make them 
>>> understand more of Cassandra than absolutely critical to use it", I lean:
>>> 
>>> 1. DENSE_VECTOR
>>> 2. VECTOR
>>> 3. type[dimension]
>>> 
>>> This leaves the path open for us to expand on it in the future with sparse 
>>> support and allows us to introduce some semantics that indicate idioms 
>>> around nullability for the users coming from a different space.
>>> 
>>> "NON-NULL FROZEN" is strictly correct, however it requires 
>>> understanding idioms of how Cassandra thinks about data (nulls mean 
>>> different things to us, we have differences between frozen and non-frozen 
>>> due to constraints in our storage engine and materialization of data, etc) 
>>> that get in the way of users doing things in the pattern they're familiar 
>>> with without learning more about the DB than they're probably looking to 
>>> learn. Historically this has been a challenge for us in adoption; the 
>>> classic "Why can't I just write and delete and write as much as I want? Why 
>>> are deletes filling up my disk?" problem comes to mind.
>>> 
>>> I'd also be happy with us supporting:
>>> * NON-NULL FROZEN
>>> * DENSE_VECTOR as syntactic sugar for the above
>>> 
>>> If getting into the "built-in syntactic sugar mapping for communities and 
>>> specific use-cases" is something we're willing to consider.
>>> 
>>> On Fri, May 5, 2023, at 7:26 AM, Patrick McFadin wrote:
>>>> I think we are still discussing implementation here when I'm talking about 
>>>> developer experience. I want developers to adopt this quickly, easily and 
>>>> be successful. Vector search is already a thing. People use it every day. 
>>>> A successful outcome, in my view, is developers picking up this feature 
>>>> without reading a manual. (Because they don't anyway and get in trouble) I 
>>>> did some more extensive research about what other DBs are using for 
>>>> syntax. The consensus is some variety of 'VECTOR', 'DENSE' and 'SPARSE'
>>>> 
>>>> Pinecone[1] - dense_vector, sparse_vector
>>>> Elastic[2]: dense_vector
>>>> 

Re: [POLL] Vector type for ML

[David Capwell]

did some more extensive research about what other DBs are using for syntax. 
>>> The consensus is some variety of 'VECTOR', 'DENSE' and 'SPARSE'
>>> 
>>> Pinecone[1] - dense_vector, sparse_vector
>>> Elastic[2]: dense_vector
>>> Milvus[3]: float_vector, binary_vector
>>> pgvector[4]: vector
>>> Weaviate[5]: Different approach. All typed arrays can be indexed
>>> 
>>> Based on that I'm advocating a similar syntax:
>>> 
>>> - DENSE VECTOR
>>> or
>>> - VECTOR
>>> 
>>> [1] https://docs.pinecone.io/docs/hybrid-search
>>> [2] 
>>> https://www.elastic.co/guide/en/elasticsearch/reference/current/dense-vector.html
>>> [3] https://milvus.io/docs/create_collection.md
>>> [4] https://github.com/pgvector/pgvector
>>> [5] https://weaviate.io/developers/weaviate/config-refs/datatypes
>>> 
>>> On Fri, May 5, 2023 at 6:07 AM Mike Adamson >> <mailto:madam...@datastax.com>> wrote:
>>> Then we can have the indexing apparatus only accept frozen for 
>>> the HSNW case.
>>> I'm inclined to agree with Benedict that the index will need to be 
>>> specifically select by option rather than inferred based on type. As such 
>>> there is no real reason for the frozen requirement on the type. The hnsw 
>>> index can be built just as easily from a non-frozen array.
>>> 
>>> I am in favour of enforcing non-null on the elements of an array by 
>>> default. I would prefer that allowing nulls in the array would be a later 
>>> addition if and when a use case arose for it.
>>> 
>>> On Fri, 5 May 2023 at 03:02, Caleb Rackliffe >> <mailto:calebrackli...@gmail.com>> wrote:
>>> Even in the ML case, sparse can just mean zeros rather than nulls, and they 
>>> should compress similarly anyway.
>>> 
>>> If we really want null values, I'd rather leave that in collections space.
>>> 
>>> On Thu, May 4, 2023 at 8:59 PM Caleb Rackliffe >> <mailto:calebrackli...@gmail.com>> wrote:
>>> I actually still prefer type[dimension], because I think I intuitively read 
>>> this as a primitive (meaning no null elements) array. Then we can have the 
>>> indexing apparatus only accept frozen for the HSNW case.
>>> 
>>> If that isn't intuitive to anyone else, I don't really have a strong 
>>> opinion...but...conflating "frozen" and "dense" seems like a bad idea. One 
>>> should indicate single vs. multi-cell, and the other the presence or 
>>> absence of nulls/zeros/whatever.
>>> 
>>> On Thu, May 4, 2023 at 12:51 PM Patrick McFadin >> <mailto:pmcfa...@gmail.com>> wrote:
>>> I agree with David's reasoning and the use of DENSE (and maybe eventually 
>>> SPARSE). This is terminology well established in the data world, and it 
>>> would lead to much easier adoption from users. VECTOR is close, but I can 
>>> see having to create a lot of content around "How to use it and not get in 
>>> trouble." (I have a lot of that content already)
>>> 
>>>  - We don't have to explain what it is. A lot of prior art out there 
>>> already [1][2][3]
>>>  - We're matching an established term with what users would expect. No 
>>> surprises. 
>>>  - Shorter ramp-up time for users. Cassandra is being modernized.
>>> 
>>> The implementation is flexible, but the interface should empower our users 
>>> to be awesome. 
>>> 
>>> Patrick
>>> 
>>> 1 - 
>>> https://stats.stackexchange.com/questions/266996/what-do-the-terms-dense-and-sparse-mean-in-the-context-of-neural-networks
>>>  
>>> <https://urldefense.com/v3/__https://stats.stackexchange.com/questions/266996/what-do-the-terms-dense-and-sparse-mean-in-the-context-of-neural-networks__;!!PbtH5S7Ebw!dpAaXazB6qZfr_FdkU9ThEq4X0DDTa-DlNvF5V4AvTiZSpHeYn6zqhFD4ZVaRLYoQBmNTn7n6jt5ymZs5Ud6ieKGQw$>
>>> 2 - 
>>> https://induraj2020.medium.com/what-are-sparse-features-and-dense-features-8d1746a77035
>>>  
>>> <https://urldefense.com/v3/__https://induraj2020.medium.com/what-are-sparse-features-and-dense-features-8d1746a77035__;!!PbtH5S7Ebw!dpAaXazB6qZfr_FdkU9ThEq4X0DDTa-DlNvF5V4AvTiZSpHeYn6zqhFD4ZVaRLYoQBmNTn7n6jt5ymZs5Ue1o2CO2Q$>
>>> 3 - 
>>> https://revware.net/sparse-vs-dense-data-the-power-of-points-and-clouds/ 
>>> <https://urldefense.com/v3/__https://revware.net/sparse-vs-dense-data-the-power-of-points-and-clouds/__;!!PbtH5S7Ebw!dpAaXazB6qZfr_FdkU9ThEq4X0DDTa-

Re: [POLL] Vector type for ML

[David Capwell]

My views have changed over time on syntax and I feel type[dimention] may not be 
the best, so it has gone lower in my own personal ranking… this is my current 
preference

1) DENSE [dimention] | NON NULL [dimention]
2) VECTOR
3) type[dimention]

My reasoning for this order

* type[dimention] looks like syntax sugar for array, so users 
may assume list/array semantics, but we limit to non-null elements in a frozen 
array
* feel VECTOR as a prefix feels out of place, but VECTOR as a direct type makes 
more sense… this also leads to a possible future of VECTOR which is the 
non-fixed length version of this type.  What makes VECTOR different from 
list/array?  non-null elements and is frozen.  I don’t feel that VECTOR really 
tells users to expect non-null or frozen semantics, as there exists different 
VECTOR types for those reasons (sparse vs dense)… 
* DENSE may be confusing for people coming from languages where this just means 
“sequential layout”, which is what our frozen array/list already are… but since 
the target user is coming from a ML background, this shouldn’t offer much 
confusion.  DENSE just means FROZEN in Cassandra, with NON NULL elements 
(SPARSE allows for NULL and isn’t frozen)… So DENSE just acts as syntax sugar 
for frozen


> On May 4, 2023, at 4:13 AM, Brandon Williams  wrote:
> 
> 1. VECTOR
> 2. VECTOR FLOAT[n]
> 3. FLOAT[N]   (Non null by default)
> 
> Redundant or not, I think having the VECTOR keyword helps signify what
> the app is generally about and helps get buy-in from ML stakeholders.
> 
> On Thu, May 4, 2023 at 3:45 AM Benedict  wrote:
>> 
>> Hurrah for initial agreement.
>> 
>> For syntax, I think one option was just FLOAT[N]. In VECTOR FLOAT[N], VECTOR 
>> is redundant - FLOAT[N] is fully descriptive by itself. I don’t think VECTOR 
>> should be used to simply imply non-null, as this would be very unintuitive. 
>> More logical would be NONNULL, if this is the only condition being applied. 
>> Alternatively for arrays we could default to NONNULL and later introduce 
>> NULLABLE if we want to permit nulls.
>> 
>> If the word vector is to be used it makes more sense to make it look like a 
>> list, so VECTOR as here the word VECTOR is clearly not redundant.
>> 
>> So, I vote:
>> 
>> 1) (NON NULL) FLOAT[N]
>> 2) FLOAT[N]   (Non null by default)
>> 3) VECTOR
>> 
>> 
>> 
>> On 4 May 2023, at 08:52, Mick Semb Wever  wrote:
>> 
>> 
>>> 
>>> Did we agree on a CQL syntax?
>>> 
>>> I don’t believe there has been a pool on CQL syntax… my understanding 
>>> reading all the threads is that there are ~4-5 options and non are -1ed, so 
>>> believe we are waiting for majority rule on this?
>> 
>> 
>> 
>> Re-reading that thread, IIUC the valid choices remaining are…
>> 
>> 1. VECTOR FLOAT[n]
>> 2. FLOAT VECTOR[n]
>> 3. VECTOR
>> 4. VECTOR[n]
>> 5. ARRAY
>> 6. NON-NULL FROZEN
>> 
>> 
>> Yes I'm putting my preference (1) first ;) because (banging on) if the 
>> future of CQL will have FLOAT[n] and FROZEN, where the VECTOR 
>> keyword is: for general cql users; just meaning "non-null and frozen", these 
>> gel best together.
>> 
>> Options (5) and (6) are for those that feel we can and should provide this 
>> type without introducing the vector keyword.
>> 
>> 

Re: [POLL] Vector type for ML

[David Capwell]

efense.com/v3/__http://cassandra.link/__;!!PbtH5S7Ebw!dpYGB0YPi3Klzqfi_uM7eXJnrZD-GoRc6HOvP3_-p7v8ya3jEKpmpkFbDSguLwD26V9VHlFOMzhMWVy7cvvbbpb74g$>
>>>>  : The best resources for Apache Cassandra
>>>> 
>>>> 
>>>> On Tue, May 2, 2023 at 6:39 PM Patrick McFadin >>> <mailto:pmcfa...@gmail.com>> wrote:
>>>>> \o/
>>>>> 
>>>>> Bring it in team. Group hug. 
>>>>> 
>>>>> Now if you'll excuse me, I'm going to go build my preso on how Cassandra 
>>>>> is the only distributed database you can do vector search in an ACID 
>>>>> transaction. 
>>>>> 
>>>>> Patrick
>>>>> 
>>>>> On Tue, May 2, 2023 at 3:27 PM Jonathan Ellis >>>> <mailto:jbel...@gmail.com>> wrote:
>>>>>> I had a call with David.  We agreed that we want a "vector" data type 
>>>>>> with these properties
>>>>>> 
>>>>>> - Fixed length
>>>>>> - No nulls
>>>>>> - Random access not supported
>>>>>> 
>>>>>> Where we disagreed was on my proposal to restrict vectors to only 
>>>>>> numeric data.  David's points were that
>>>>>> 
>>>>>> (1) He has a use case today for a data type with the other vector 
>>>>>> properties,
>>>>>> (2) It doesn't seem reasonable to create two data types with the same 
>>>>>> properties, one of which is restricted to numerics, and
>>>>>> (3) The restrictions that I want for numeric vectors make more sense at 
>>>>>> the index and function level, than at the type level.
>>>>>> 
>>>>>> I'm ready to concede that David has the better case here and move 
>>>>>> forward with a vector implementation without that restriction.
>>>>>> 
>>>>>> On Tue, May 2, 2023 at 4:03 PM David Capwell >>>>> <mailto:dcapw...@apple.com>> wrote:
>>>>>>>>  How about it, David? Did you already make this?
>>>>>>> 
>>>>>>> I checked out the patch, fixed serialize/deserialize, added the 
>>>>>>> constraints, then added a composeForFloat(ByteBuffer), with this the 
>>>>>>> impact to the POC patch was the following
>>>>>>> 
>>>>>>> 1) move away from VectorType.instance.serializer().deserialize(bb) to 
>>>>>>> type.composeForFloat(bb), both return float[]
>>>>>>> 2) change the index validate logic to move away from checking 
>>>>>>> VectorType and instead check for that plus the element type == 
>>>>>>> FloatType.  I didn’t bother to do this as its trivial
>>>>>>> 
>>>>>>>> David. End this argument. SHOW THE CODE! 
>>>>>>> 
>>>>>>> If this argument ends and people are cool with vector supporting 
>>>>>>> abstract type, more than glad to help get this in.
>>>>>>> 
>>>>>>>> On May 2, 2023, at 1:53 PM, Jeremy Hanna >>>>>>> <mailto:jeremy.hanna1...@gmail.com>> wrote:
>>>>>>>> 
>>>>>>>> I'm all for bringing more functionality to the masses sooner, but the 
>>>>>>>> original idea has a very very specific use case.  Do we have use cases 
>>>>>>>> for a general purpose Vector/Array data structure?  If so, awesome.  I 
>>>>>>>> just wondered if generalizing provides value, beyond being 
>>>>>>>> straightforward to implement.  I'm just trying to be sensitive to the 
>>>>>>>> database code maintenance and driver support for general types versus 
>>>>>>>> a single type for a specific, well defined purpose.
>>>>>>>> 
>>>>>>>> If it could easily be a plugin, that's great - but the full picture 
>>>>>>>> involves drivers that need to support it or you end up getting binary 
>>>>>>>> blobs you have to decode client side and then do stuff with.  So 
>>>>>>>> ideally if you have a well defined use case that you can build into 
>>>>>>>> the database, having it just be part of the database and associated 
>>>>>>>> drivers - that makes the experience much much better.
>>>>

Re: [POLL] Vector type for ML

[David Capwell]

>  How about it, David? Did you already make this?

I checked out the patch, fixed serialize/deserialize, added the constraints, 
then added a composeForFloat(ByteBuffer), with this the impact to the POC patch 
was the following

1) move away from VectorType.instance.serializer().deserialize(bb) to 
type.composeForFloat(bb), both return float[]
2) change the index validate logic to move away from checking VectorType and 
instead check for that plus the element type == FloatType.  I didn’t bother to 
do this as its trivial

> David. End this argument. SHOW THE CODE! 

If this argument ends and people are cool with vector supporting abstract type, 
more than glad to help get this in.

> On May 2, 2023, at 1:53 PM, Jeremy Hanna  wrote:
> 
> I'm all for bringing more functionality to the masses sooner, but the 
> original idea has a very very specific use case.  Do we have use cases for a 
> general purpose Vector/Array data structure?  If so, awesome.  I just 
> wondered if generalizing provides value, beyond being straightforward to 
> implement.  I'm just trying to be sensitive to the database code maintenance 
> and driver support for general types versus a single type for a specific, 
> well defined purpose.
> 
> If it could easily be a plugin, that's great - but the full picture involves 
> drivers that need to support it or you end up getting binary blobs you have 
> to decode client side and then do stuff with.  So ideally if you have a well 
> defined use case that you can build into the database, having it just be part 
> of the database and associated drivers - that makes the experience much much 
> better.
> 
> I'm not trying to say B couldn't be valuable or that a plugin couldn't be 
> feasible.  I'm just trying to enlarge the picture a bit to see what that 
> means for this use case and for the supporting drivers/clients.
> 
>> On May 2, 2023, at 3:04 PM, Benedict  wrote:
>> 
>> But it’s so trivial it was already implemented by David in the span of ten 
>> minutes? If anything, we’re slowing progress down by refusing to do the 
>> extra types, as we’re busy arguing about it rather than delivering a feature?
>> 
>> FWIW, my interpretation of the votes today is that we SHOULD NOT (ever) 
>> support types beyond float. Not that we should start with float.
>> 
>> So, this whole debate is a mess, I think. But hey ho.
>> 
>>> On 2 May 2023, at 20:57, Patrick McFadin  wrote:
>>> 
>>> 
>>> I'll speak up on that one. If you look at my ranked voting, that is where 
>>> my head is. I get accused of scope creep (a lot) and looking at the initial 
>>> proposal Jonathan put on the ML it was mostly "Developers are adopting 
>>> vector search at a furious pace and I think I have a simple way of adding 
>>> support to keep Cassandra relevant for these use cases" Instead of just 
>>> focusing on this use case, I feel the arguments have bike shedded into 
>>> scope creep which means it will take forever to get into the project.
>>> 
>>> My preference is to see one thing validated with an MVP and get it into the 
>>> hands of developers sooner so we can continue to iterate based on actual 
>>> usage. 
>>> 
>>> It doesn't say your points are wrong or your opinions are broken, I'm 
>>> voting for what I think will be awesome for users sooner. 
>>> 
>>> Patrick
>>> 
>>> On Tue, May 2, 2023 at 12:29 PM Benedict >> > wrote:
 Could folk voting against a general purpose type (that could well be 
 called a vector) briefly explain their reasoning?
 
 We established in the other thread that it’s technically trivial, meaning 
 folk must think it is strictly superior to only support float rather than 
 eg all numeric types (note: for the type, not the ANN). 
 
 I am surprised, and the blurbs accompanying votes so far don’t seem to 
 touch on this, mostly just endorsing the idea of a vector.
 
 
> On 2 May 2023, at 20:20, Patrick McFadin  > wrote:
> 
> 
> A > B > C on both polls. 
> 
> Having talked to several users in the community that are highly excited 
> about this change, this gets to what developers want to do at Cassandra 
> scale: store embeddings and retrieve them. 
> 
> On Tue, May 2, 2023 at 11:47 AM Andrés de la Peña  > wrote:
>> A > B > C
>> 
>> I don't think that ML is such a niche application that it can't have its 
>> own CQL data type. Also, vectors are mathematical elements that have 
>> more applications that ML.
>> 
>> On Tue, 2 May 2023 at 19:15, Mick Semb Wever > > wrote:
>>> 
>>> 
>>> On Tue, 2 May 2023 at 17:14, Jonathan Ellis >> > wrote:
 Should we add a vector type to Cassandra designed to meet the needs of 
 machine learning use cases, specifically feature and embedding vectors 
 for training, inference, 

Re: [POLL] Vector type for ML

[David Capwell]

> B) Should we introduce a type that is general purpose, and supports all 
> Cassandra types, so that this may be used to support ML (and perhaps other) 
> workloads

I vote B only as well...

> On May 2, 2023, at 9:02 AM, Benedict  wrote:
> 
> This is not the poll I thought we would be conducting, and I don’t really 
> support its framing. There are two parallel questions: what the functionality 
> should be and how they should be exposed. This poll compresses the 
> optionality poorly.
> 
> Whether or not we support a “vector” concept (or something isomorphic with 
> it), the first question this poll wants to answer is:
> 
> A) Should we introduce a new CQL collection type that is unique to ML and 
> *only* supports float32
> B) Should we introduce a type that is general purpose, and supports all 
> Cassandra types, so that this may be used to support ML (and perhaps other) 
> workloads
> C) Should we not introduce new types to CQL at all
> 
> For this question, I vote B only.
> 
> Once this question is answered it makes sense to answer how it will be 
> exposed semantically/syntactically. 
> 
> 
>> On 2 May 2023, at 16:43, Jonathan Ellis  wrote:
>> 
>> 
>> My preference: A > B > C.  Vectors are distinct enough from arrays that we 
>> should not make adding the latter a prerequisite for adding the former.
>> 
>> On Tue, May 2, 2023 at 10:13 AM Jonathan Ellis > > wrote:
>>> Should we add a vector type to Cassandra designed to meet the needs of 
>>> machine learning use cases, specifically feature and embedding vectors for 
>>> training, inference, and vector search?  
>>> 
>>> ML vectors are fixed-dimension (fixed-length) sequences of numeric types, 
>>> with no nulls allowed, and with no need for random access. The ML industry 
>>> overwhelmingly uses float32 vectors, to the point that the industry-leading 
>>> special-purpose vector database ONLY supports that data type.
>>> 
>>> This poll is to gauge consensus subsequent to the recent discussion thread 
>>> at https://lists.apache.org/thread/0lj1nk9jbhkf1rlgqcvxqzfyntdjrnk0.
>>> 
>>> Please rank the discussed options from most preferred option to least, 
>>> e.g., A > B > C (A is my preference, followed by B, followed by C) or C > B 
>>> = A (C is my preference, followed by B or A approximately equally.)
>>> 
>>> (A) I am in favor of adding a vector type for floats; I do not believe we 
>>> need to tie it to any particular implementation details.
>>> 
>>> (B) I am okay with adding a vector type but I believe we must add array 
>>> types that compose with all Cassandra types first, and make vectors a 
>>> special case of arrays-without-null-elements.
>>> 
>>> (C) I am not in favor of adding a built-in vector type.
>>> 
>>> -- 
>>> Jonathan Ellis
>>> co-founder, http://www.datastax.com 
>>> @spyced
>> 
>> 
>> -- 
>> Jonathan Ellis
>> co-founder, http://www.datastax.com 
>> @spyced

Re: [DISCUSS] New data type for vector search

[David Capwell]

> A data type plug-in is actually really easy today, I think? 

Sadly not, the client reads the class from our schema tables and has to have 
duplicate logic to serialize/deserialize results… types are easy to add if you 
are ok with client not understanding them (and will some clients fail due to 
every language having its own logic?)

> On May 1, 2023, at 2:26 PM, Benedict  wrote:
> 
> A data type plug-in is actually really easy today, I think? But, developing 
> further hooks should probably be thought through as they’re necessary. 
> 
> I think in this case it would be simpler to deliver a general purpose type, 
> which is why I’m trying to propose types that would be acceptable.
> 
> I also think we’re pretty close to agreement, really?
> 
> But if not, let’s flesh out potential plug-in requirements.
> 
> 
>> On 1 May 2023, at 21:58, Josh McKenzie  wrote:
>> 
>> 
>>> 
>>> If we want to make an ML-specific data type, it should be in an ML plug-in.
>> How can we encourage a healthier plug-in ecosystem? As far as I know it's 
>> been pretty anemic historically:
>> 
>> cassandra: 
>> https://cassandra.apache.org/doc/latest/cassandra/plugins/index.html
>> postgres: https://www.postgresql.org/docs/current/contrib.html
>> 
>> I'm really interested to hear if there's more in the ecosystem I'm not aware 
>> of or if there's been strides made in this regard; users in the ecosystem 
>> being able to write durable extensions to Cassandra that they can then 
>> distribute and gain momentum could potentially be a great incubator for new 
>> features or functionality in the ecosystem.
>> 
>> If our support for extensions remains as bare as I believe it to be, I 
>> wouldn't recommend anyone go that route.
>> 
>> On Mon, May 1, 2023, at 4:17 PM, Benedict wrote:
>>> 
>>> I have explained repeatedly why I am opposed to ML-specific data types. If 
>>> we want to make an ML-specific data type, it should be in an ML plug-in. We 
>>> should not pollute the general purpose language with hastily-considered 
>>> features that target specific bandwagons - at best partially - no matter 
>>> how exciting the bandwagon.
>>> 
>>> I think a simple and easy case can be made for fixed length array types 
>>> that do not seem to create random bits of cruft in the language that dangle 
>>> by themselves should this play not pan out. This is an easy way for this 
>>> effort to make progress without negatively impacting the language.
>>> 
>>> That is, unless we want to start supporting totally random types for every 
>>> use case at the top level language layer. I don’t think this is a good 
>>> idea, personally, and I’m quite confident we would now be regretting this 
>>> approach had it been taken for earlier bandwagons.
>>> 
>>> Nor do I think anyone’s priors about how successful this effort will be 
>>> should matter. As a matter of principle, we should simply never deliver a 
>>> specialist functionality as a high level CQL language feature without at 
>>> least baking it for several years as a plug-in.
>>> 
>>>> On 1 May 2023, at 21:03, Mick Semb Wever  wrote:
>>>> 
>>>> 
>>>> Yes!  What you (David) and Benedict write beautifully supports `VECTOR 
>>>> FLOAT[n]` imho.
>>>> 
>>>> You are definitely bringing up valid implementation details, and that can 
>>>> be dealt with during patch review. This thread is about the CQL API 
>>>> addition.  
>>>> 
>>>> No matter which way the technical review goes with the implementation 
>>>> details, `VECTOR FLOAT[n]` does not limit it, and gives us the most ML 
>>>> idiomatic approach and the best long-term CQL API.  It's a win-win 
>>>> situation – no matter how you look at it imho it is the best solution api 
>>>> wise.  
>>>> 
>>>> Unless the suggestion is that an ideal implementation can give us a better 
>>>> CQL API – but I don't see what that could be.   Maybe the suggestion is we 
>>>> deny the possibility of using the VECTOR keyword and bring us back to 
>>>> something like `NON-NULL FROZEN`.   This is odd to me because 
>>>> `VECTOR` here can be just an alias for `NON-NULL FROZEN` while meeting the 
>>>> patch's audience and their idioms.  I have no problems with introducing 
>>>> such an alias to meet the ML crowd.
>>>> 
>>>> Another way I think of this is
>>>>  `VECTOR FLOAT[n]` is the por

Re: [DISCUSS] New data type for vector search

[David Capwell]

> I think a simple and easy case can be made for fixed length array types that 
> do not seem to create random bits of cruft in the language that dangle by 
> themselves should this play not pan out. 

If I am understanding you correctly, then a "VECTOR FLOAT[n]” is fine as its a 
array type but has 2 new properties: fixed length, NON NULL elements… for 
optimization reasons we can always have a special composeForFloat(ByteBuffer) 
-> float[] which can be used by ANN (avoids boxing and java.util.List)

This makes sure the type doesn’t care about ML and can be used for other use 
cases, but for any ML systems that come after the fact they can special for 
this type… this is basically the change I made locally to test out the effort

> On May 1, 2023, at 1:17 PM, Benedict  wrote:
> 
> I have explained repeatedly why I am opposed to ML-specific data types. If we 
> want to make an ML-specific data type, it should be in an ML plug-in. We 
> should not pollute the general purpose language with hastily-considered 
> features that target specific bandwagons - at best partially - no matter how 
> exciting the bandwagon.
> 
> I think a simple and easy case can be made for fixed length array types that 
> do not seem to create random bits of cruft in the language that dangle by 
> themselves should this play not pan out. This is an easy way for this effort 
> to make progress without negatively impacting the language.
> 
> That is, unless we want to start supporting totally random types for every 
> use case at the top level language layer. I don’t think this is a good idea, 
> personally, and I’m quite confident we would now be regretting this approach 
> had it been taken for earlier bandwagons.
> 
> Nor do I think anyone’s priors about how successful this effort will be 
> should matter. As a matter of principle, we should simply never deliver a 
> specialist functionality as a high level CQL language feature without at 
> least baking it for several years as a plug-in.
> 
>> On 1 May 2023, at 21:03, Mick Semb Wever  wrote:
>> 
>> 
>> 
>> Yes!  What you (David) and Benedict write beautifully supports `VECTOR 
>> FLOAT[n]` imho.
>> 
>> You are definitely bringing up valid implementation details, and that can be 
>> dealt with during patch review. This thread is about the CQL API addition.  
>> 
>> No matter which way the technical review goes with the implementation 
>> details, `VECTOR FLOAT[n]` does not limit it, and gives us the most ML 
>> idiomatic approach and the best long-term CQL API.  It's a win-win situation 
>> – no matter how you look at it imho it is the best solution api wise.  
>> 
>> Unless the suggestion is that an ideal implementation can give us a better 
>> CQL API – but I don't see what that could be.   Maybe the suggestion is we 
>> deny the possibility of using the VECTOR keyword and bring us back to 
>> something like `NON-NULL FROZEN`.   This is odd to me because 
>> `VECTOR` here can be just an alias for `NON-NULL FROZEN` while meeting the 
>> patch's audience and their idioms.  I have no problems with introducing such 
>> an alias to meet the ML crowd.
>> 
>> Another way I think of this is
>>  `VECTOR FLOAT[n]` is the porcelain ML cql api,
>>  `NON-NULL FROZEN` and `FROZEN` and `FLOAT[n]` are the 
>> general-use plumbing cql apis. 
>> 
>> This would allow implementation details to be moved out of this thread and 
>> to the review phase.
>> 
>> 
>> 
>> 
>> On Mon, 1 May 2023 at 20:57, David Capwell > <mailto:dcapw...@apple.com>> wrote:
>>> > I think it is totally reasonable that the ANN patch (and Jonathan) is not 
>>> > asked to implement on top of, or towards, other array (or other) new data 
>>> > types.
>>> 
>>> 
>>> This impacts serialization, if you do not think about this day 1 you then 
>>> can’t add later on without having to worry about migration and versioning… 
>>> 
>>> Honestly I wanted to better understand the cost to be generic and the 
>>> impact to ANN, so I took 
>>> https://github.com/jbellis/cassandra/blob/vsearch/src/java/org/apache/cassandra/db/marshal/VectorType.java
>>>  and made it handle every requirement I have listed so far (size, null, all 
>>> types)… the current patch has several bugs at the type level that would 
>>> need to be fixed, so had to fix those as well…. Total time to do this was 
>>> 10 minutes… and this includes adding a method "public float[] 
>>> composeAsFloats(ByteBuffer bytes)” which made the change to existing logic 
>>> small (change VectorType.Serializer.instance.de

Re: [DISCUSS] New data type for vector search

[David Capwell]

> I think it is totally reasonable that the ANN patch (and Jonathan) is not 
> asked to implement on top of, or towards, other array (or other) new data 
> types.


This impacts serialization, if you do not think about this day 1 you then can’t 
add later on without having to worry about migration and versioning… 

Honestly I wanted to better understand the cost to be generic and the impact to 
ANN, so I took 
https://github.com/jbellis/cassandra/blob/vsearch/src/java/org/apache/cassandra/db/marshal/VectorType.java
 and made it handle every requirement I have listed so far (size, null, all 
types)… the current patch has several bugs at the type level that would need to 
be fixed, so had to fix those as well…. Total time to do this was 10 minutes… 
and this includes adding a method "public float[] composeAsFloats(ByteBuffer 
bytes)” which made the change to existing logic small (change 
VectorType.Serializer.instance.deserialize(buffer) to 
type.composeAsFloats(buffer))….

Did this have any impact to the final ByteBuffer?  Nope, it had identical 
layout for the FloatType case, but works for all types…. I didn’t change the 
fact we store the size (felt this could be removed, but then we could never 
support expanding the vector in the future…)

So, given the fact it takes a few minutes to implement all these requirements, 
I do find it very reasonable to push back and say we should make sure the new 
type is not leaking details from a special ANN index…. We have spent more time 
debating this than it takes to support… we also have fuzz testing on trunk so 
just updating org.apache.cassandra.utils.AbstractTypeGenerators to know about 
this new type means we get type coverage as well…

I have zero issues helping to review this patch and make sure the testing is 
on-par with existing types (this is a strong requirement for me)


> On May 1, 2023, at 10:40 AM, Mick Semb Wever  wrote:
> 
> 
> > But suggesting that Jonathan should work on implementing general purpose 
> > arrays seems to fall outside the scope of this discussion, since the result 
> > of such work wouldn't even fill the need Jonathan is targeting for here. 
> 
> Every comment I have made so far I have argued that the v1 work doesn’t need 
> to do some things, but that the limitations proposed so far are not real 
> requirements; there is a big difference between what “could be allowed” and 
> what is implemented day one… I am pushing back on what “could be allowed”, so 
> far every justification has been that it slows down the ANN work…
> 
> Simple examples of this already exists in C* (every example could be enhanced 
> logically, we just have yet to put in the work)
> 
> * updating a element of a list is only allowed for multi-cell
> * appending to a list is only allowed for multi-cell
> * etc.
> 
> By saying that the type "shall not support", you actively block future work 
> and future possibilities...
> 
> 
> 
> I am coming around strongly to the `VECTOR FLOAT[n]` option.
> 
> This gives Jonathan the simplest path right now with ths ANN work, while also 
> ensuring the CQL API gets the best future potential.
> 
> With `VECTOR FLOAT[n]` the 'vector' is the ml sugar that means non-null and 
> frozen, and that allows both today and in the future, as desired, for its 
> implementation to be entirely different to `FLOAT[n]`.  This addresses a 
> number of people's concerns that we meet ML's idioms head on.
> 
> IMHO it feels like it will fit into the ideal future CQL , where all 
> `primitive[N]` are implemented, and where we have VECTOR FLOAT[n] (and maybe 
> VECTOR BYTE[n]). This will also permit in the future `FROZEN` 
> if we wanted nulls in frozen arrays.
> 
> I think it is totally reasonable that the ANN patch (and Jonathan) is not 
> asked to implement on top of, or towards, other array (or other) new data 
> types.
> 
> I also think it is correct that we think about the evolution of CQL's API,  
> and how it might exist in the future when we have both ml vectors and general 
> use arrays.

Re: [DISCUSS] New data type for vector search

[David Capwell]

supports floats *even though 
>>> postgresql already has an array data type* because the semantics are 
>>> different.  Random access doesn't make sense, string and collection and 
>>> other datatypes don't make sense, typical ordered indexes don't make sense, 
>>> etc.  It's just a different beast from arrays, for a different use case.
>>> 
>>> On Fri, Apr 28, 2023 at 10:40 AM Benedict  wrote:
>>> But you’re proposing introducing a general purpose type - this isn’t an ML 
>>> plug-in, it’s modifying the core language in a manner that makes targeting 
>>> your workload easier. Which is fine, but that means you have to consider 
>>> its impact on the general language, not just your target use case.
>>> 
>>>> On 28 Apr 2023, at 16:29, Jonathan Ellis  wrote:
>>>> 
>>>> That's exactly right.
>>>> 
>>>> In particular it makes no sense at all from an ML perspective to have 
>>>> vector types of anything other than numerics.  And as I mentioned in the 
>>>> POC thread (but I did not mention here), float is overwhelmingly the most 
>>>> frequently used vector type, to the point that Pinecone (by far the most 
>>>> popular vector search engine) ONLY supports that type.
>>>> 
>>>> Lucene and Elastic also add support for vectors of bytes (8-bit ints), 
>>>> which are useful for optimizing models that you have already built with 
>>>> floats, but we have no reasonable path towards supporting indexing and 
>>>> searches against any other vector type.
>>>> 
>>>> So in order of what makes sense to me:
>>>> 
>>>> 1. Add a vector type for just floats; consider adding bytes later if 
>>>> demand materializes. This gives us 99% of the value and limits the scope 
>>>> so we can deliver quickly.
>>>> 
>>>> 2. Add a vector type for floats or bytes. This gives us another 1% of 
>>>> value in exchange for an extra 20% or so of effort.
>>>> 
>>>> 3. Add a vector type for all numeric primitives, but you can only index 
>>>> floats and bytes.  I think this is confusing to users and a bad idea.
>>>> 
>>>> 4. Add a vector type that composes with all Cassandra types.  I can't see 
>>>> a reason to do this, nobody wants it, and we killed the most similar 
>>>> proposal in the past as wontfix.
>>>> 
>>>> On Thu, Apr 27, 2023 at 7:49 PM Josh McKenzie  wrote:
>>>> From a machine learning perspective, vectors are a well-known concept that 
>>>> are effectively immutable fixed-length n-dimensional values that are then 
>>>> later used either as part of a model or in conjunction with a model after 
>>>> the fact.
>>>> 
>>>> While we could have this be non-frozen and not call it a vector, I'd be 
>>>> inclined to still make the argument for a layer of syntactic sugar on top 
>>>> that met ML users where they were with concepts they understood rather 
>>>> than forcing them through the cognitive lift of figuring out the Cassandra 
>>>> specific contortions to replicate something that's ubiquitous in their 
>>>> space. We did the same "Cassandra-first" approach with our JSON support 
>>>> and that didn't do us any favors in terms of adoption and usage as far as 
>>>> I know.
>>>> 
>>>> So is the goal here to provide something specific and idiomatic for the ML 
>>>> community or is the goal to make a primitive that's C*-centric that then 
>>>> another layer can write to? I personally argue for the former; I don't see 
>>>> this specific data type going away any time soon.
>>>> 
>>>> On Thu, Apr 27, 2023, at 12:39 PM, David Capwell wrote:
>>>>>> but as you point out it has the problem of allowing nulls.
>>>>> 
>>>>> If nulls are not allowed for the elements, then either we need  a) a new 
>>>>> type, or b) add some way to say elements may not be null…. As much as I 
>>>>> do like b, I am leaning towards new type for this use case.
>>>>> 
>>>>> So, to flesh out the type requirements I have seen so far
>>>>> 
>>>>> 1) represents a fixed size array of element type
>>>>> * on write path we will need to validate this
>>>>> 2) element may not be null
>>>>> * on write path we will need to validate this
>>>>> 3) “frozen” (is th

Re: [DISCUSS] New data type for vector search

[David Capwell]

> but as you point out it has the problem of allowing nulls.

If nulls are not allowed for the elements, then either we need  a) a new type, 
or b) add some way to say elements may not be null…. As much as I do like b, I 
am leaning towards new type for this use case.

So, to flesh out the type requirements I have seen so far

1) represents a fixed size array of element type
* on write path we will need to validate this
2) element may not be null
* on write path we will need to validate this
3) “frozen” (is this really a requirement for the type or is this just simpler 
for the ANN work?  I feel that this shouldn’t be a requirement)
4) works for all types (my requirement; original proposal is float only, but 
could logically expand to primitive types)

Anything else?

> The key thing about a vector is that unlike lists or tuples you really don't 
> care about individual elements, you care about doing vector and matrix 
> multiplications with the thing as a unit. 

That maybe true for this use case, but “should” this be true for the type 
itself?  I feel like no… if a user wants the Nth element of a vector why would 
we block them?  I am not saying the first patch, or even 5.0 adds support for 
index access, I am just trying to push back saying that the type should not 
block this.

> (Maybe this is making the case for VECTOR FLOAT[N] rather than FLOAT 
> VECTOR[N].)

Now that nulls are not allowed, I have mixed feelings about FLOAT[N], I prefer 
this syntax but that limitation may not be desired for all use cases… we could 
always add LIST and ARRAY later to address that case.

In terms of syntax I have seen, here is my ordered preference:

1) TYPE[size] - have mixed feelings due to non-null, but still prefer it
2) QUALIFIER TYPE[size] - QUALIFIER is just a Term we use to denote this 
semantic…. Could even be NON NULL TYPE[size]

> On Apr 27, 2023, at 9:00 AM, Benedict  wrote:
> 
> That’s a bounded ring buffer, not a fixed length array.
> 
> This definitely isn’t a tuple because the types are all the same, which is 
> pretty crucial for matrix operations. Matrix libraries generally work on 
> arrays of known dimensionality, or sparse representations.
> 
> Whether we draw any semantic link between the frozen list and whatever we do 
> here, it is fundamentally a frozen list with a restriction on its size. What 
> we’re defining here are “statically” sized arrays, whereas a frozen list is 
> essentially a dynamically sized array.
> 
> I do not think vector is a good name because vector is used in some other 
> popular languages to mean a (dynamic) list, which is confusing when we also 
> have a list concept.
> 
> I’m fine with just using the FLOAT[N] syntax, and drawing no direct link with 
> list. Though it is a bit strange that this particular type declaration looks 
> so different to other collection types.
> 
>> On 27 Apr 2023, at 16:48, Jeff Jirsa  wrote:
>> 
>> 
>> 
>> 
>> On Thu, Apr 27, 2023 at 7:39 AM Jonathan Ellis > > wrote:
>>> It's been a while, so I may be missing something, but do we already have 
>>> fixed-size lists?  If not, I don't see why we'd try to make this fit into a 
>>> List-shaped problem.
>> 
>> We do not. The proposal got closed as wont-fix  
>> https://issues.apache.org/jira/browse/CASSANDRA-9110
>> 
>> 

Re: [DISCUSS] New data type for vector search

[David Capwell]

Benedicts comments also makes me question; can any of the values in the vector 
be null?  The patch sent works with float arrays, so null isn’t possible… is 
null not valid for a vector type?  If so this would help justify why is a 
vector not a array or a list (both allow null)

> On Apr 26, 2023, at 10:50 AM, David Capwell  wrote:
> 
> Thanks for starting this thread!
> 
>> In the initial commits and thread, this was DENSE FLOAT32. Nobody really 
>> loved that, so we considered a bunch of alternatives, including
>> 
>> - `FLOAT[N]`: This minimal option resembles C and Java array syntax, which 
>> would make it familiar for many users. However, this syntax raises the 
>> question of why arrays cannot be created for other types.  Additionally, the 
>> expectation for an array is to provide random access to its contents, which 
>> is not supported for vectors.
>> - `DENSE FLOAT[N]`: This option clarifies that we are supporting dense 
>> vectors, not sparse ones. However, since Lucene had sparse vector support in 
>> the past but removed it for lack of compelling use cases, it is unlikely 
>> that it will be added back, making the "DENSE" qualifier less relevant.
>> - `DENSE FLOAT VECTOR[N]`: This is the most verbose option and aligns with 
>> the CQL/SQL spirit. However, the "DENSE" qualifier is unnecessary for the 
>> reasons mentioned above.
>> - `VECTOR FLOAT[N]`: This option omits the "DENSE" qualifier, but has a less 
>> natural word order.
>> `VECTOR`: This follows the syntax of our Collections, but again 
>> this would imply that random access is supported, which we want to avoid 
>> doing.
>> - `VECTOR[N]`: This syntax is not very clear about the vector's contents and 
>> could make it difficult to add other vector types, such as byte vectors 
>> (already supported by Lucene), in the future.
> 
> I didn’t look close enough when I saw your patch, is this type multicell or 
> not?  Aka is this acting like a frozen> of fixed size?  I had 
> assumed its non-multicell…. Main reason I ask this now is this pushback for 
> random access…. Lets say I have the following table
> 
> CREATE TABLE fluffy_kittens (
>   pk int PRIMARY KEY,
>   vector FLOAT[42] — don’t ask why fluffy kittens need a vector, they just do!
> )
> 
> If I do the following query, I would expect it to work
> 
> SELECT vector[7] FROM fluffy_kittens WHERE pk=0; — 7 is less than 42
> 
> While working on accord’s CQL integration Caleb and I kept getting bitten by 
> frozen vs non frozen behavior, so many cases just stopped working on frozen 
> collections and should be easy to add (we force user to load the full value 
> already, why can we not touch it?).
> 
> Now, back to the random access comment, assuming this is not multicell why 
> would random access be blocked?  If the type isValueLengthFixed() == true 
> then random access should be simple (else it does require walking the array 
> in-order or to fully deserialize the BB (if working with Lucene I assume we 
> already deserialized out of BB)).  I am just trying to flesh out if there is 
> a limitation not being brought up or is this trying to limit the scope of 
> access for easier testing?
> 
>> However, this syntax raises the question of why arrays cannot be created for 
>> other types
> 
> Left this comment in the other thread, why not?  This could be useful outside 
> the float use case, so having a new "VectorType(AbstractType elements, int 
> size)” is easier/better than a float only version.  I also did a lot of work 
> to fuzz test our type system, so just adding that into the existing generator 
> would get good coverage right off the bat (have another fuzz tester I have 
> not contributed yet, it was done for Accord… it fuzz tests the AST, so would 
> be easy to add this there, that would test type specific access, which the 
> existing tests don’t)
> 
>> Finally, the original qualifier of 32 in `FLOAT32` was intended to allow 
>> consistency if we add other float types like FLOAT16 or FLOAT64
> 
> I do not think we should add a new FLOAT32 type, but I am cool with an alias 
> that has FLOAT32 point to FLOAT.  One negative of this is that the code paths 
> where we return schema back to users would do FLOAT even if user wrote 
> FLOAT32… other than that negative I don’t see any other problems.
> 
>> Thus, we believe that `FLOAT VECTOR[N_DIMENSIONS]` provides the best balance 
>> of clarity, conciseness, and extensibility. It is more natural in its word 
>> order than the original proposal and avoids unnecessary qualifiers, while 
>> still being clear about the data type it represents. Finally, this syntax is 
&

Re: [DISCUSS] New data type for vector search

[David Capwell]

Thanks for starting this thread!

> In the initial commits and thread, this was DENSE FLOAT32. Nobody really 
> loved that, so we considered a bunch of alternatives, including
> 
> - `FLOAT[N]`: This minimal option resembles C and Java array syntax, which 
> would make it familiar for many users. However, this syntax raises the 
> question of why arrays cannot be created for other types.  Additionally, the 
> expectation for an array is to provide random access to its contents, which 
> is not supported for vectors.
> - `DENSE FLOAT[N]`: This option clarifies that we are supporting dense 
> vectors, not sparse ones. However, since Lucene had sparse vector support in 
> the past but removed it for lack of compelling use cases, it is unlikely that 
> it will be added back, making the "DENSE" qualifier less relevant.
> - `DENSE FLOAT VECTOR[N]`: This is the most verbose option and aligns with 
> the CQL/SQL spirit. However, the "DENSE" qualifier is unnecessary for the 
> reasons mentioned above.
> - `VECTOR FLOAT[N]`: This option omits the "DENSE" qualifier, but has a less 
> natural word order.
> `VECTOR`: This follows the syntax of our Collections, but again 
> this would imply that random access is supported, which we want to avoid 
> doing.
> - `VECTOR[N]`: This syntax is not very clear about the vector's contents and 
> could make it difficult to add other vector types, such as byte vectors 
> (already supported by Lucene), in the future.

I didn’t look close enough when I saw your patch, is this type multicell or 
not?  Aka is this acting like a frozen> of fixed size?  I had 
assumed its non-multicell…. Main reason I ask this now is this pushback for 
random access…. Lets say I have the following table

CREATE TABLE fluffy_kittens (
  pk int PRIMARY KEY,
  vector FLOAT[42] — don’t ask why fluffy kittens need a vector, they just do!
)

If I do the following query, I would expect it to work

SELECT vector[7] FROM fluffy_kittens WHERE pk=0; — 7 is less than 42

While working on accord’s CQL integration Caleb and I kept getting bitten by 
frozen vs non frozen behavior, so many cases just stopped working on frozen 
collections and should be easy to add (we force user to load the full value 
already, why can we not touch it?).

Now, back to the random access comment, assuming this is not multicell why 
would random access be blocked?  If the type isValueLengthFixed() == true then 
random access should be simple (else it does require walking the array in-order 
or to fully deserialize the BB (if working with Lucene I assume we already 
deserialized out of BB)).  I am just trying to flesh out if there is a 
limitation not being brought up or is this trying to limit the scope of access 
for easier testing?

> However, this syntax raises the question of why arrays cannot be created for 
> other types

Left this comment in the other thread, why not?  This could be useful outside 
the float use case, so having a new "VectorType(AbstractType elements, int 
size)” is easier/better than a float only version.  I also did a lot of work to 
fuzz test our type system, so just adding that into the existing generator 
would get good coverage right off the bat (have another fuzz tester I have not 
contributed yet, it was done for Accord… it fuzz tests the AST, so would be 
easy to add this there, that would test type specific access, which the 
existing tests don’t)

> Finally, the original qualifier of 32 in `FLOAT32` was intended to allow 
> consistency if we add other float types like FLOAT16 or FLOAT64

I do not think we should add a new FLOAT32 type, but I am cool with an alias 
that has FLOAT32 point to FLOAT.  One negative of this is that the code paths 
where we return schema back to users would do FLOAT even if user wrote FLOAT32… 
other than that negative I don’t see any other problems.

> Thus, we believe that `FLOAT VECTOR[N_DIMENSIONS]` provides the best balance 
> of clarity, conciseness, and extensibility. It is more natural in its word 
> order than the original proposal and avoids unnecessary qualifiers, while 
> still being clear about the data type it represents. Finally, this syntax is 
> straighforwardly extensible should we choose to support other vector types in 
> the future.

My preference is TYPE[n_dimension] but I am ok with this syntax if others 
prefer it.  I don’t agree that this extra verbosity adds more clarity, there 
seems to be an assumption that this will tell users that random access isn’t 
allowed and only blessed types are allowed… both points I feel are not valid 
(or not seen anything published why they should be valid).  There is a 
difference between what a type “could” do and what we implement day 1, I 
wouldn’t want to add more verbosity because of intentions of the day 1 
implementation. 


> On Apr 26, 2023, at 7:31 AM, Jonathan Ellis  wrote:
> 
> Hi all,
> 
> Splitting this out per the suggestion in the initial VS thread so we can work 
> on driver support in parallel with the 

Re: Adding vector search to SAI with heirarchical navigable small world graph index

[David Capwell]

> DENSE seems to just be an array? So very similar to a frozen list, but with a 
> fixed size?

How I read the doc, DENSE = ARRAY, but knew that couldn’t be the case, so when 
I read the code its fixed size array…. So the real syntax was “DENSE 
FLOAT32[42]”

Not a fan of the type naming, and feel that a fixed size array could be useful 
for other cases as well, so think we can improve here (personally prefer 
float[42], text[42], etc… vector maybe closer to our existing syntax 
but not a fan).

> I guess this is an excellent example to explore the minima of what 
> constitutes a CEP

The ANN change itself feels like a CEP makes sense.  Are we going to depend on 
Lucene’s HNSW or build our own?  How do we validate this for correctness?  What 
does correctness mean in a distributed context?  Is this going to be pluggable 
(big push recently to offer plugability)?


> On Apr 26, 2023, at 7:37 AM, Patrick McFadin  wrote:
> 
> I guess this is an excellent example to explore the minima of what 
> constitutes a CEP. So far, CEPs have been some large changes, so where does 
> something like this fit? (Wait. Did I beat Benedict to a Bike Shed? I think I 
> did.)
> 
> This is a list of everything needed for a CEP:
> 
> Status
> Scope
> Goals
> Approach
> Timeline
> Mailing list / Slack channels
> Related JIRA tickets
> Motivation
> Audience
> Proposed Changes
> New or Changed Public Interfaces
> Compatibility, Deprecation, and Migration Plan
> Test Plan
> Rejected Alternatives
> 
> This is a big enough change to provide information for each element. Going 
> back to the spirit of why we started CEPs, we wanted to avoid a mega-commit 
> without some shaping and agreement before code goes into trunk. I don't have 
> a clear indication of where that line lies. From our own wiki: "It is highly 
> recommended to pursue a CEP for significant user-facing or changes that cut 
> across multiple subsystems." That seems to fit here. Part of my motivation is 
> being clear with potential new contributors by example and encouraging more 
> awesomeness.  
> 
> The changes for operators:
>  - New drivers
>  - New gaurdrails?
>  - Indexing == storage requirements
> 
> Patrick
> 
> On Tue, Apr 25, 2023 at 10:53 PM Mick Semb Wever  wrote:
> I was soo happy when I saw this, I know many users are going to be 
> thrilled about it.
> 
> 
> On Wed, 26 Apr 2023 at 05:15, Patrick McFadin  wrote:
> Not sure if this is what you are saying, Josh, but I believe this needs to be 
> its own CEP. It's a change in CQL syntax and changes how clusters operate. 
> The change needs to be documented and voted on. Jonathan, you know how to 
> find me if you want me to help write it. :) 
> 
> I'd be fine with just a DISCUSS thread to agree to the CQL change, since it: 
> `DENSE FLOAT32` appears to be a minimal,  and the overall patch building on 
> SAI. As Henrik mentioned there's other SAI extensions being added too without 
> CEPs.  Can you elaborate on how you see this changing how the cluster 
> operates?
> 
> This will be easier to decide once we have a patch to look at, but that 
> depends on a CEP-7 base (e.g. no feature branch exists). If we do want a CEP 
> we need to allow a few weeks to get it through, but that can happen in 
> parallel and maybe drafting up something now will be valuable anyway for an 
> eventual CEP that proposes the more complete features (e.g. 
> cosine_similarity(…)). 
> 

Re: Adding vector search to SAI with heirarchical navigable small world graph index

[David Capwell]

This work sounds interesting, I would recommend decoupling the types from the 
ANN support as the types require client changes and can go in now (would give a 
lot of breathing room to get this ready for 5.0), where as ANN depends on SAI 
which is still being worked on.

> On Apr 22, 2023, at 1:02 PM, Jonathan Ellis  wrote:
> 
> If you're interested in playing with HNSW outside of a super alpha Cassandra 
> branch, I put up a repo with some sample code here: 
> https://github.com/jbellis/hnswdemo
> 
> On Fri, Apr 21, 2023 at 4:19 PM Jonathan Ellis  > wrote:
>> Happy Friday, everyone!
>> 
>> Rich text formatting ahead, I've attached a PDF for those who prefer that.
>> 
>> I propose adding approximate nearest neighbor (ANN) vector search capability 
>> to Apache Cassandra via storage-attached indexes (SAI). This is a 
>> medium-sized effort that will significantly enhance Cassandra’s 
>> functionality, particularly for AI use cases. This addition will not only 
>> provide a new and important feature for existing Cassandra users, but also 
>> attract new users to the community from the AI space, further expanding 
>> Cassandra’s reach and relevance.
>> Introduction
>> Vector search is a powerful document search technique that enables 
>> developers to quickly find relevant content within an extensive collection 
>> of documents, which is useful as a standalone technique, but it is 
>> particularly hot now because it significantly enhances the performance of 
>> LLMs.
>> 
>> Vector search uses ML models to match the semantics of a question rather 
>> than just the words it contains, avoiding the classic false positives and 
>> false negatives associated with term-based search.  Alessandro Benedetti 
>> gives some good examples in his excellent talk 
>> :
>> 
>> 
>> 
>> 
>> 
>> You can search across any set of vectors, which are just ordered sets of 
>> numbers.  In the context of natural language queries and document search, we 
>> are specifically concerned with a type of vector called an embedding.  
>> 
>> An embedding is a high-dimensional vector that captures the underlying 
>> semantic relationships and contextual information of words or phrases. 
>> Embeddings are generated by ML models trained for this purpose; OpenAI 
>> provides an API to do this, but open-source and self-hostable models like 
>> BERT are also popular. Creating more accurate and smaller embeddings are 
>> active research areas in ML.
>> 
>> Large language models (LLMs) can be described as a mile wide and an inch 
>> deep. They are not experts on any narrow domain (although they will 
>> hallucinate that they are, sometimes convincingly).  You can remedy this by 
>> giving the LLM additional context for your query, but the context window is 
>> small (4k tokens for GPT-3.5, up to 32k for GPT-4), so you want to be very 
>> selective about giving the LLM the most relevant possible information.
>> 
>> Vector search is red-hot now because it allows us to easily answer the 
>> question “what are the most relevant documents to provide as context” by 
>> performing a similarity search between the embeddings vector of the query, 
>> and those of your document universe.  Doing exact search is prohibitively 
>> expensive, since you necessarily have to compare with each and every 
>> document; this is intractable when you have billions or trillions of docs.  
>> However, there are well-understood algorithms for turning this into a 
>> logarithmic problem if you are willing to accept approximately the most 
>> similar documents.  This is the “approximate nearest neighbor” problem.  
>> (You will see these referred to as kNN – k nearest neighbors – or ANN.)
>> 
>> Pinecone DB has a good example of what this looks like in Python code 
>> .
>> 
>> Vector search is the foundation underlying effectively all of the AI 
>> applications that are launching now.  This is particularly relevant to 
>> Apache Cassandra users, who tend to manage the types of large datasets that 
>> benefit the most from fast similarity search. Adding vector search to 
>> Cassandra’s unique strengths of scale, reliability, and low latency, will 
>> further enhance its appeal and effectiveness for these users while also 
>> making it more attractive to newcomers looking to harness AI’s potential.  
>> The faster we deliver vector search, the more valuable it will be for this 
>> expanding user base.
>> Requirements
>> Perform vector search as outlined in the Pinecone example above
>> Support Float32 embeddings in the form of a new DENSE FLOAT32 cql type
>> This is also useful for “classic” ML applications that derive and serve 
>> their own feature vectors
>> Add ANN (approximate nearest neighbor) search.
>> Work with normal Cassandra data flow
>> Inserting one row at a time is fine; cannot require batch ingest
>> Updating, deleting rows is also fine
>> Must 

Re: [DISCUSS] CEP-29 CQL NOT Operator

[David Capwell]

Overall I welcome this feature, was trying to use this around 1-2 months back 
and found we didn’t support, so glad to see it coming!

From a testing point of view, I think we would want to have good fuzz testing 
covering complex types (frozen/non-frozen collections, tuples, udt, etc.), and 
reverse ordering; both sections tend to cause the most problem for new features 
(and existing ones)

We also will want a way to disable this feature, and optionally disable at 
different sections (such as m2’s NOT IN for partition keys).

> On Apr 4, 2023, at 2:28 AM, Piotr Kołaczkowski  wrote:
> 
> Hi everyone!
> 
> I created a new CEP for adding NOT support to the query language and
> want to start discussion around it:
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-29%3A+CQL+NOT+operator
> 
> Happy to get your feedback.
> --
> Piotr

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[David Capwell]

Assuming we do not release with it, then yes, as we wouldn’t need to maintain.  
My point for this case was that I don’t feel the time cost is worth it, I am 
not -1 if someone wants to add, was more saying our time is better off else 
where.

We currently don’t touch Transactional Metadata, we have custom logic (which 
relies on client to tell every instance in the cluster an update happened) that 
we are using right now in tests and deployed clusters.  So once we can 
integrate with Transactional Metadata, we stop relying on clients to tell us 
about topology changes… a custom/thrown away epoch generator would make this 
current process more user friendly, but would need to spend time to make sure 
stable when deployed on a cluster

> On Mar 24, 2023, at 12:43 PM, Josh McKenzie  wrote:
> 
>> If this is in a release, we then need to maintain that feature, so would be 
>> against it.
> Isn't the argument that cep-21 provides this so we could just remove the 
> temporary impl and point to the new facility for this generation?
> 
> On Fri, Mar 24, 2023, at 3:22 PM, David Capwell wrote:
>>> the question we want to answer is whether or not we build a throwaway patch 
>>> for linearizable epochs
>> 
>> If this is in a release, we then need to maintain that feature, so would be 
>> against it.
>> 
>> If this is for testing, then I would argue the current world is “fine”… 
>> current world is hard to use and brittle (users need to tell accord that the 
>> cluster changed), but if accord is rebasing on txn metadata then this won’t 
>> be that way long (currently blocked from doing that due to txn metadata not 
>> passing all tests yet).
>> 
>>> On Mar 24, 2023, at 12:12 PM, Josh McKenzie  wrote:
>>> 
>>>> FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration 
>>>> risk while accepting the possible delivery risk.
>>> What does the chain of rebases against trunk look like here? cep-21-tcm 
>>> rebase, then cep-15 on cep-21-tcm, then cep-7 on cep-21-tcm, then a race on 
>>> whichever of 15 or 7 merge after 21 goes into trunk? Or 7 based on 15, the 
>>> other way around...
>>> 
>>> I'm not actively working on any of these branches so take my perspective 
>>> with that appropriate grain of salt, but the coupling of these things seems 
>>> to have it's own kind of breed of integration pain to upkeep over time 
>>> depending on how frequently we're rebasing against trunk.
>>> 
>>>> the question we want to answer is whether or not we build a throwaway 
>>>> patch for linearizable epochs
>>> Do we have an informed opinion on how long we think this would take? Seems 
>>> like that'd help clarify whether or not there's contributors with the 
>>> bandwidth and desire to even do that or whether everyone depending on 
>>> cep-21 is our option.
>>> 
>>> On Fri, Mar 24, 2023, at 1:30 PM, Caleb Rackliffe wrote:
>>>> I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm 
>>>> here: https://github.com/apache/cassandra/pull/2227
>>>> 
>>>> It wasn't too terrible, and I was actually able to get the main CQL-based 
>>>> Accord tests working as long as I disabled automatic forwarding of CAS and 
>>>> SERIAL read operations to Accord. The bigger issue was test stability in 
>>>> cep-21-tcm. I'm sure that will mature quickly here, and I created a few 
>>>> issues to attach to the Transactional Metadata epic 
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18330>.
>>>> 
>>>> In the meantime, I rebased cep-15-accord on trunk at commit 
>>>> 3eb605b4db0fa6b1ab67b85724a9cfbf00aae7de. The option to finish the 
>>>> remaining bits of CASSANDRA-18196 
>>>> <https://issues.apache.org/jira/browse/CASSANDRA-18196> and merge w/o TCM 
>>>> is still available, but it sounds like the question we want to answer is 
>>>> whether or not we build a throwaway patch for linearizable epochs in lieu 
>>>> of TCM?
>>>> 
>>>> FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration 
>>>> risk while accepting the possible delivery risk.
>>>> 
>>>> On Fri, Mar 24, 2023 at 9:32 AM Josh McKenzie >>> <mailto:jmcken...@apache.org>> wrote:
>>>> 
>>>>> making sure that joining and leaving nodes update some state via Paxos 
>>>>> instead of via gossip
>>>> What kind of a time delivery risk does coupling CEP-15 with CEP-21 
>>>> introduce (i.e. unk-un

Re: [DISCUSS] cep-15-accord, cep-21-tcm, and trunk

[David Capwell]

> the question we want to answer is whether or not we build a throwaway patch 
> for linearizable epochs

If this is in a release, we then need to maintain that feature, so would be 
against it.

If this is for testing, then I would argue the current world is “fine”… current 
world is hard to use and brittle (users need to tell accord that the cluster 
changed), but if accord is rebasing on txn metadata then this won’t be that way 
long (currently blocked from doing that due to txn metadata not passing all 
tests yet).

> On Mar 24, 2023, at 12:12 PM, Josh McKenzie  wrote:
> 
>> FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk 
>> while accepting the possible delivery risk.
> What does the chain of rebases against trunk look like here? cep-21-tcm 
> rebase, then cep-15 on cep-21-tcm, then cep-7 on cep-21-tcm, then a race on 
> whichever of 15 or 7 merge after 21 goes into trunk? Or 7 based on 15, the 
> other way around...
> 
> I'm not actively working on any of these branches so take my perspective with 
> that appropriate grain of salt, but the coupling of these things seems to 
> have it's own kind of breed of integration pain to upkeep over time depending 
> on how frequently we're rebasing against trunk.
> 
>> the question we want to answer is whether or not we build a throwaway patch 
>> for linearizable epochs
> Do we have an informed opinion on how long we think this would take? Seems 
> like that'd help clarify whether or not there's contributors with the 
> bandwidth and desire to even do that or whether everyone depending on cep-21 
> is our option.
> 
> On Fri, Mar 24, 2023, at 1:30 PM, Caleb Rackliffe wrote:
>> I actually did a dry run rebase of cep-15-accord on top of cep-21-tcm here: 
>> https://github.com/apache/cassandra/pull/2227
>> 
>> It wasn't too terrible, and I was actually able to get the main CQL-based 
>> Accord tests working as long as I disabled automatic forwarding of CAS and 
>> SERIAL read operations to Accord. The bigger issue was test stability in 
>> cep-21-tcm. I'm sure that will mature quickly here, and I created a few 
>> issues to attach to the Transactional Metadata epic 
>> .
>> 
>> In the meantime, I rebased cep-15-accord on trunk at commit 
>> 3eb605b4db0fa6b1ab67b85724a9cfbf00aae7de. The option to finish the remaining 
>> bits of CASSANDRA-18196 
>>  and merge w/o TCM is 
>> still available, but it sounds like the question we want to answer is 
>> whether or not we build a throwaway patch for linearizable epochs in lieu of 
>> TCM?
>> 
>> FWIW, I'd still rather just integrate w/ TCM ASAP, avoiding integration risk 
>> while accepting the possible delivery risk.
>> 
>> On Fri, Mar 24, 2023 at 9:32 AM Josh McKenzie > > wrote:
>> 
>>> making sure that joining and leaving nodes update some state via Paxos 
>>> instead of via gossip
>> What kind of a time delivery risk does coupling CEP-15 with CEP-21 introduce 
>> (i.e. unk-unk on CEP-21 leading to delay cascades to CEP-15)? Seems like 
>> having a table we CAS state for on epochs wouldn't be too challenging, but 
>> I'm not familiar w/the details so I could be completely off here.
>> 
>> Being able to deliver both of these things on their own timetable seems like 
>> a pretty valuable thing assuming the lift required would be modest.
>> 
>> On Fri, Mar 24, 2023, at 6:15 AM, Benedict wrote:
>>> 
>>> Accord doesn’t have a hard dependency on CEP-21 fwiw, it just needs 
>>> linearizable epochs. This could be achieved with a much more modest patch, 
>>> essentially avoiding almost all of the insertion points of cep-21, just 
>>> making sure that joining and leaving nodes update some state via Paxos 
>>> instead of via gossip, and assign an epoch as part of the update.
>>> 
>>> It would be preferable to use cep-21 since it introduces this 
>>> functionality, and our intention is to use cep-21 for this. But it isn’t a 
>>> hard dependency.
>>> 
>>> 
 On 22 Mar 2023, at 20:58, Henrik Ingo >>> > wrote:
 
 Since Accord depends on transactional meta-data... is there really any 
 alternative than what you propose?
 
 Sure, if there is some subset of Accord that could be merged, while work 
 continues on a branched that is based on CEP-21 branch, that would be 
 great. Merging even a prototype of Accord to trunk probably has marketing 
 value. (Don't laugh, many popular databases have had "atomic transactions, 
 except if anyone executes DDL simultaneously".)
 
 On Tue, Mar 14, 2023 at 8:39 PM Caleb Rackliffe >>> > wrote:
 We've already talked a bit 
 
  about how and when the current Accord feature branch should merge to 
 trunk. Earlier today, 

Re: Role of Hadoop code in Cassandra 5.0

[David Capwell]

Isn’t our deprecation rules that if we deprecate in 4.0.0 we can remove in 5.x, 
but 4.x needs to wait for 6.x?  I am cool deprecating this and willing to pull 
into another repo if people (not me) are willing to maintain it (else just 
delete).

> On Mar 10, 2023, at 1:13 AM, Jacek Lewandowski  
> wrote:
> 
> I've experimentally added 
> https://issues.apache.org/jira/browse/CASSANDRA-16984 to 
> https://issues.apache.org/jira/browse/CASSANDRA-18306 (post 4.0 cleanup)
> 
> - - -- --- -  -
> Jacek Lewandowski
> 
> 
> pt., 10 mar 2023 o 09:56 Berenguer Blasi  > napisał(a):
>> +1 deprecate + removal
>> 
>> On 10/3/23 1:41, Jeremy Hanna wrote:
>>> It was mainly to integrate with Hadoop - I used it from 0.6 to 1.2 in 
>>> production prior to starting at DataStax and at that time I was stitching 
>>> together Cloudera's distribution of Hadoop with Cassandra.  Back then there 
>>> were others that used it as well.  As far as I know, usage dropped off when 
>>> the Spark Cassandra Connector got pretty mature.  It enabled people to take 
>>> an off the shelf Hadoop distribution and run the Hadoop processes on the 
>>> same nodes or external to the Cassandra cluster and get topology 
>>> information to do things like Hadoop splits and things like that through 
>>> the Hadoop interfaces.  I think the version lag is an indication that it 
>>> hasn't been used recently.  Also, like others have said, the Spark 
>>> Cassandra Connector is really what people should be using at this point 
>>> imo.  That or depending on the use case, Apple's bulk reader: 
>>> https://github.com/jberragan/spark-cassandra-bulkreader that is mentioned 
>>> on https://issues.apache.org/jira/browse/CASSANDRA-16222.
>>> 
 On Mar 9, 2023, at 12:00 PM, Rahul Xavier Singh 
   wrote:
 
 What is the hadoop code for? For interacting from Hadoop via CQL, or 
 Thrift if it's that old, or directly looking at SSTables? Been using C* 
 since 2 and have never used it. 
 
 Agree to deprecate in next possible 4.1.x version and remove in 5.0 
 
 Rahul Singh
 Chief Executive Officer | Business Platform Architect
 m: 202.905.2818 e: rahul.si...@anant.us  li: 
 http://linkedin.com/in/xingh ca: http://calendly.com/xingh
 
 We create, support, and manage real-time global data & analytics platforms 
 for the modern enterprise.
 
 Anant | https://anant.us 
  3 Washington Circle, Suite 301
 Washington, D.C. 20037
 
 http://Cassandra.Link  : The best resources for 
 Apache Cassandra
 
 
 On Thu, Mar 9, 2023 at 12:53 PM Brandon Williams >>> > wrote:
> I think if we reach consensus here that decides it. I too vote to
> deprecate in 4.1.x.  This means we would remove it in 5.0.
> 
> Kind Regards,
> Brandon
> 
> On Thu, Mar 9, 2023 at 11:32 AM Ekaterina Dimitrova
> mailto:e.dimitr...@gmail.com>> wrote:
> >
> > Deprecation sounds good to me, but I am not completely sure in which 
> > version we can do it. If it is possible to add a deprecation warning in 
> > the 4.x series or at least 4.1.x - I vote for that.
> >
> > On Thu, 9 Mar 2023 at 12:14, Jacek Lewandowski 
> > mailto:lewandowski.ja...@gmail.com>> 
> > wrote:
> >>
> >> Is it possible to deprecate it in the 4.1.x patch release? :)
> >>
> >>
> >> - - -- --- -  -
> >> Jacek Lewandowski
> >>
> >>
> >> czw., 9 mar 2023 o 18:11 Brandon Williams  >> > napisał(a):
> >>>
> >>> This is my feeling too, but I think we should accomplish this by
> >>> deprecating it first.  I don't expect anything will change after the
> >>> deprecation period.
> >>>
> >>> Kind Regards,
> >>> Brandon
> >>>
> >>> On Thu, Mar 9, 2023 at 11:09 AM Jacek Lewandowski
> >>> mailto:lewandowski.ja...@gmail.com>> 
> >>> wrote:
> >>> >
> >>> > I vote for removing it entirely.
> >>> >
> >>> > thanks
> >>> > - - -- --- -  -
> >>> > Jacek Lewandowski
> >>> >
> >>> >
> >>> > czw., 9 mar 2023 o 18:07 Miklosovic, Stefan 
> >>> >  >>> > > napisał(a):
> >>> >>
> >>> >> Derek,
> >>> >>
> >>> >> I have couple more points ... I do not think that extracting it to 
> >>> >> a separate repository is "win". That code is on Hadoop 1.0.3. We 
> >>> >> would be spending a lot of work on extracting it just to extract 
> >>> >> 10 years old code with occasional updates (in my humble opinion 
> >>> >> just to make it compilable again if the code around changes). What 
> >>> >> good is in that? We would have one more place to take care of ... 
> >>> >> 

Re: hsqldb test dependency in simulator code

[David Capwell]

Quickly looking I think we can switch to 
org.agrona.collections.Long2LongHashMap, the key isn’t the “correct” type (long 
when we want int) but isn’t too hard to switch.  Few differences in the 
semantics need to be handled, but not much

1) get of non-defeiled key should throw NoSuchElementException
2) get with long[] updates the array and returns boolean; easy enough to port 
(copy signature/semantics or just change the single caller)
3) org.hsqldb.store.BaseHashMap#addOrRemove(long, long, java.lang.Object, 
java.lang.Object, boolean) for us is just a put

> On Mar 14, 2023, at 5:32 AM, Derek Chen-Becker  wrote:
> 
> Do we have the hsqldb library just for a primitive oriented equivalent of 
> Map? If this is test code only that seems like a pretty easy thing 
> to replace and I would vote to implement that so we can fully drop the 
> dependency. 
> 
> Cheers,
> 
> Derek
> 
> On Tue, Mar 14, 2023 at 5:49 AM Miklosovic, Stefan 
> mailto:stefan.mikloso...@netapp.com>> wrote:
>> Hi list,
>> 
>> while removing Hadoop code in trunk, as agreed on ML recently, I did that 
>> but we need to do this (1). By removing all Hadoop dependencies, we also 
>> removed hsqldb library we surprisingly rely on so I am putting it back.
>> 
>> Honestly, I do not know if people are even aware of the fact we are 
>> depending on hsqldb in simulator package. Can not we rewrite that so we can 
>> drop this dependency entirely?
>> 
>> Do we want to spend time on rewriting this and then we remove Hadoop 
>> libraries or I just add this library among the test ones?
>> 
>> Thanks
>> 
>> (1) https://github.com/apache/cassandra/pull/2212/files#r1133841928
> 
> 
> -- 
> +---+
> | Derek Chen-Becker |
> | GPG Key available at https://keybase.io/dchenbecker and   |
> | https://pgp.mit.edu/pks/lookup?search=derek%40chen-becker.org |
> | Fngrprnt: EB8A 6480 F0A3 C8EB C1E7  7F42 AFC5 AFEE 96E4 6ACC  |
> +---+
> 

Re: [DISCUSS] Next release date

[David Capwell]

I am cool with defining target release date and working backwards from there.  
If we do want to go this route, I think we do need to answer why 4.1 cut -> 
release took so much time, and if people could start validation “before” we 
branch?  If we know trunk is stable today then we could release today, but I 
don’t believe we have this level of testing today, so I don’t know if I could 
say we can release in 1-4 months.

> On Mar 1, 2023, at 9:21 AM, J. D. Jordan  wrote:
> 
> We have been talking a lot about the branch cutting date, but I agree with 
> Benedict here, I think we should actually be talking about the expected 
> release date. 
> 
> If we truly believe that we can release within 1-2 months of cutting the 
> branch, and many people I have talked to think that is possible, then a May 
> branch cut means we release by July. That would only be 7 months post 4.1 
> release, that seems a little fast to me.  IIRC the last time we had release 
> cadence discussions most people were for keeping to a release cadence of 
> around 12 months, and many were against a 6 month cadence.
> 
> So if we want to have a goal of “around 12 months” and also have a goal of 
> “release before summit in December”. I would suggest we put our release date 
> goal in October to give some runway for being late and still getting out by 
> December.
> 
> So if the release date goal is October, we can also hedge with the longer 2 
> month estimate on “time after branching” to again make sure we make our 
> goals. This would put the branching in August. So if we do release in an 
> October that gives us 10 months since 4.1, which while still shorter than 12 
> much closer than only 7 months would be.
> 
> If people feel 1 month post branch cut is feasible we could cut the branch in 
> September.
> 
> -Jeremiah
> 
>> On Mar 1, 2023, at 10:34 AM, Henrik Ingo  wrote:
>> 
>> 
>> Hi 
>> 
>> Those are great questions Mick. It's good to recognize this discussion 
>> impacts a broad range of contributors and users, and not all of them might 
>> be aware of the discussion in the first place.
>> 
>> More generally I would say that your questions brought to mind two 
>> fundamental principles with a "train model" release schedule:
>> 
>>   1. If a feature isn't ready by the cut-off date, there's no reason to 
>> delay the release, because the next release is guaranteed to be just around 
>> the corner.
>>   2. If there is a really important feature that won't make it, rather than 
>> delaying the planned release, we should (also) consider the opposite: we can 
>> do the next release earlier if there is a compelling feature ready to go. 
>> (Answers question 2b from Mick.)
>> 
>> I have arguments both for and against moving the release date:
>> 
>> 
>> The to stick with the current plan, is that we have a lot of big features 
>> now landing in trunk. If we delay the release for one feature, it will delay 
>> the GA of all the other features that were ready by May. For example, while 
>> SAI code is still being massaged based on review comments, we fully expect 
>> it to merge before May. Same for the work on tries, which is on its final 
>> stretch. Arguably Java 17 support can't come soon enough either. And so 
>> on... For some user it can be a simple feature, like just one specific 
>> guardrail, that they are waiting on. So just as we are excited to wait for 
>> that one feature to be ready and make it, I'm just as unexcited about the 
>> prospect of delaying the GA of several other features. If we had just one 
>> big feature that everyone was working on, this would be easier to decide...
>> 
>> Note also that postponing the release for a single feature that is still in 
>> development is a risky bet, because you never know what unknowns are still 
>> ahead once the work is code complete and put to more serious testing. At 
>> first it might sound reasonable to delay 1-3 months, but what if on that 3rd 
>> month some unforeseen work is discovered, and now we need to discuss 
>> delaying another 3 months. Such a risk is inherent to any software project, 
>> and we should anticipate it to happen. Scott's re-telling of CASSANDRA-18110 
>> is a great example: These delays can happen due to a single issue, and it 
>> can be hard to speed things up by e.g. assigning more engineers to the work. 
>> So, when we say that we'd like to move the branching date from May to 
>> August, and specifically in order for some feature to be ready by then, what 
>> do we do if it's not ready in August?`It's presumably closer to being ready 
>> at that point, so the temptation to wait just a little bit more is always 
>> there. (And this is also my answer to Mick's question 1b.)
>> 
>> 
>> 
>> Now, let me switch to arguing the opposite opinion:
>> 
>> My instinct here would be to stick to early May as the cut-off date, but 
>> also allow for exceptions. I'm curious to hear how this proposal is 
>> received? If this was a startup, there could be a CEO or 

Re: [DISCUSS] Allow UPDATE on settings virtual table to change running configuration

[David Capwell]

ur replies. Let me add some comments too,
> 
> 
> From a public API perspective, we have three types of fields in the
> Config class: internal use only (e.g. logger, PROPERTY_PREFIX prefix),
> read-only use (e.g. cluster_name), and read-write fields that are
> currently mutated with JMX. So a single @Mutable annotation is not
> enough to have clear Config's field separation. Adding two annotations
> @Mutable and @Immutable might solve the problem, but such an approach
> leads to code duplication if we want to extend our solution in future
> with additional parameters such as "description", besides having two
> different annotations for the same thing might confuse developers who
> are not familiar with this discussion.
> 
> So, from my point of view, the best way for us might be as follows
> mentioned in the PR (the annotation name needs to reflect that the
> fields are available to the public API and for a user, we can change
> the name):
> @Exposure(policy = Exposure.Policy.READ_WRITE)
> @Exposure(policy = Exposure.Policy.READ_ONLY)
> 
> Some other names come into my mind: APIAvailable, APIExposed,
> UserAvailable, UserExposed etc.
> 
> 
> Stefan mentioned that these annotations could be used to create
> documentation pages, it's true, I have the same thoughts in mind, and
> you can see what it will look like at the link below (the description
> annotation field will be removed from the final version of the PR, but
> may still be added as part of another issue):
> 
> https://github.com/apache/cassandra/pull/2133/files#diff-e966f41bc2a418becfe687134ec8cf542eb051eead7fb4917e65a3a2e7c9bce3R392
> 
> The SettingsTable may have the following columns and be truly
> self-descriptive for a user: name, value, default_value, policy, and
> description.
> 
> 
> Benedict mentioned that we could create a second class to hold such
> information. The best candidate for this is the ConfigFields class,
> which is based on the Config class and contains all the field names as
> constants (I used a small utility class to generate it). But it will
> still require some manual work, as there is no rule to distinguish
> which config field is mutable and which isn't. So we would have to
> update two classes instead of one (the Config class) when adding new
> configuration fields, which we don't want to do.
> 
> Here it is in the PR:
> https://github.com/apache/cassandra/pull/2133/files#diff-fcb4c5bc59d4bb127ffbe9f1ce566b2238c5bb92622da430a4ff879781093d3fR31
> 
> On Wed, 1 Mar 2023 at 09:21, Miklosovic, Stefan
>  wrote:
>> 
>> I am fine with annotations. I am not a big of fan of the generation. From my 
>> experience whenever we wanted to generate something we had to take care of 
>> the generator itself and then we had to live with what it generated (yeah, 
>> that is also a thing) instead of writing it by hand once and have some 
>> freedom to tweak it however we wanted. Splitting this into the second class 
>> ... well, I would say that just increases the entropy.
>> 
>> We can parse config class on these annotations and produce the documentation 
>> easily. I would probably go so far that I would put that annotation on all 
>> fields. We could have two - Mutable, and Immutable. But that is really 
>> optional.
>> 
>> 
>> From: Benedict 
>> Sent: Wednesday, March 1, 2023 9:09
>> To: dev@cassandra.apache.org
>> Subject: Re: [DISCUSS] Allow UPDATE on settings virtual table to change 
>> running configuration
>> 
>> NetApp Security WARNING: This is an external email. Do not click links or 
>> open attachments unless you recognize the sender and know the content is 
>> safe.
>> 
>> 
>> 
>> Another option would be to introduce a second class with the same fields as 
>> the first where we simply specify final for immutable fields, and construct 
>> it after parsing the Config.
>> 
>> We could even generate the non-final version from the one with final fields.
>> 
>> Not sure this would be nicer, but it is an alternative.
>> 
>> On 1 Mar 2023, at 02:10, Ekaterina Dimitrova  wrote:
>> 
>> 
>> I agree with David that the annotations seem a bit too many but if I have to 
>> choose from the three approaches - the annotations one seems most reasonable 
>> to me and I didn’t have the chance to consider any others. Volatile seems 
>> fragile and unclear as a differentiator. I agree
>> 
>> On Tue, 28 Feb 2023 at 17:47, Maxim Muzafarov 
>> mailto:mmu...@apache.org>> wrote:
>> Folks,
>> 
>> If there are no objections to the approach described in this thread,
>>

Re: [DISCUSS] Allow UPDATE on settings virtual table to change running configuration

[David Capwell]

I guess back to the point of the thread, we need a way to know what configs are 
mutable for the settings virtual table, so need some way to denote that the 
config replica_filtering_protection.cached_rows_fail_threshold is mutable.  
Given the way that the yaml config works, we can’t rely on the presences of 
“final” or not, so need some way to mark a config is mutable for that table, 
does anyone want to offer feedback on what works best for them?

Out of all proposals given so far “volatile” is the least verbose but also not 
explicit (as this thread is showing there is debate on if this should be 
present), new annotations are a little more verbose but would be explicit (no 
surprises), and getter/setters in different classes (such as DD) is the most 
verbose and suffers from not being explicit and ambiguity for mapping back to 
Config.   

Given the above, annotations sounds like the best option, but do we really want 
our config to look as follows?

@Replaces(oldName = "native_transport_idle_timeout_in_ms", converter = 
Converters.MILLIS_DURATION_LONG, deprecated = true)
@Mutable
public DurationSpec.LongMillisecondsBound native_transport_idle_timeout = new 
DurationSpec.LongMillisecondsBound("0ms”);
@Mutable
public DurationSpec.LongMillisecondsBound transaction_timeout = new 
DurationSpec.LongMillisecondsBound("30s”);
@Mutable
public double phi_convict_threshold = 8.0;
public String partitioner; // assume immutable by default?


> On Feb 22, 2023, at 6:20 AM, Benedict  wrote:
> 
> Could you describe the issues? Config that is globally exposed should ideally 
> be immutable with final members, in which case volatile is only necessary if 
> you’re using the config parameter in a tight loop that you need to witness a 
> new value - which shouldn’t apply to any of our config.
> 
> There are some weird niches, like updating long values on some (unsupported 
> by us) JVMs that may tear. Technically you also require it for visibility 
> with the JMM. But in practice it is mostly unnecessary. Often what seems to 
> be a volatile issue is really something else.
> 
>> On 22 Feb 2023, at 13:18, Benjamin Lerer  wrote:
>> 
>> I have seen issues with some updatable parameters which were missing the 
>> volatile keyword.
>> 
>> Le mer. 22 févr. 2023 à 11:36, Aleksey Yeshchenko  a 
>> écrit :
>> FWIW most of those volatile fields, if not in fact all of them, should NOT 
>> be volatile at all. Someone started the trend and most folks have been 
>> copycatting or doing the same for consistency with the rest of the codebase.
>> 
>> Please definitely don’t rely on that.
>> 
>>> On 21 Feb 2023, at 21:06, Maxim Muzafarov  wrote:
>>> 
>>> 1. Rely on the volatile keyword in front of fields in the Config class;
>>> 
>>> I would say this is the most confusing option for me because it
>>> doesn't give us all the guarantees we need, and also:
>>> - We have no explicit control over what exactly we expose to a user.
>>> When we modify the JMX API, we're implementing a new method for the
>>> MBean, which in turn makes this action an explicit exposure;
>>> - The volatile keyword is not the only way to achieve thread safety,
>>> and looks strange for the public API design point;
>>> - A good example is the setEnableDropCompactStorage method, which
>>> changes the volatile field, but is only visible for testing purposes;
>> 
>> 

Re: Intra-project dependencies

[David Capwell]

After a lot of effort I think this branch is in a good state, accord feels 
mostly like its in-tree and all the complexity of git is hidden mostly.  I 
would love more feedback as the patch is in a usable state

> On Jan 30, 2023, at 3:16 PM, David Capwell  wrote:
> 
> I took a stab at creating a patch that I think addresses most of the comments 
> I saw in this thread, would love feedback in 
> https://issues.apache.org/jira/browse/CASSANDRA-18204
> 
> Given that the leading solution is git submodules I went down this path and 
> fleshed out the things I saw in this thread.  I don’t think this patch is 
> 100% perfect (been trying to figure out release logic to confirm) so would 
> love to here places that I neglected or problem areas found!
> 
>> On Jan 20, 2023, at 6:48 AM, Mick Semb Wever > <mailto:m...@apache.org>> wrote:
>> 
>>  
>>>>> Both a git post-checkout and a build fail-fast will protect us here. But 
>>>>> the post-checkout will need to fail silently if the .git subdirectory 
>>>>> doesn't exist.
>>>> 
>>>> Correction: the build fail-fast will need to fail silently if the .git 
>>>> subdirectory doesn't exist.
>>> 
>>> 
>>> How will this work for users downloading source distributions?
>> 
>> It is presumed that the source found in the submodule is on the correct SHA. 
>> The integrity checks are in place when creating and when voting on the 
>> source tarball release. This means that the the build of the submodule has 
>> to be part of the in-tree build (which I have assumed already).
> 

Re: [DISCUSS] Moving system property names to the CassandraRelevantProperties

[David Capwell]

> Sure, a new checkstyle rule was added to address this case for production and 
> test classes.

Glad to hear =)

> On Feb 9, 2023, at 10:45 AM, Maxim Muzafarov  wrote:
> 
> David,
> 
>> Should” we detect this was violated and fail the build?
> 
> You are asking a good question! Sure, a new checkstyle rule was added
> to address this case for production and test classes.
> 
> On Thu, 9 Feb 2023 at 19:40, David Capwell  wrote:
>> 
>> All properties meant to be used only for tests would have a prefix like 
>> "cassandra.test.name.of.property" and production properties would be 
>> "cassandra.xyx". Once this is done, we can filter them out in vtable so 
>> there would not be any test-related properties in production. Test 
>> properties should be visible only when developing / testing Cassandra, in my 
>> opinion.
>> 
>> 
>> Good point, I wouldn’t want to expose properties meant to break C* for 
>> testing… that implies to users we should be using it!
>> 
>> I understand that there is a lot of legacy in place and we can not rename 
>> properties just like that for people.
>> 
>> 
>> We could always look to do things like we did in Config (which you call 
>> out), add a way to “migrate” to the new naming/format.  I am not sure if 
>> there is enough configs to justify this, but with 5.0 happening it may be a 
>> good time to think about normalizing these.
>> 
>> We are trying to clean up the source code around the direct use of system 
>> properties and make this use more manageable and transparent.
>> 
>> 
>> I didn’t review, but one question I have is inline with "I would like to 
>> describe the ideal state / end goal”, but from the point of view of 
>> maintaining things…. If someone adds a new system property “should” they use 
>> this enum?  “Should” we detect this was violated and fail the build?
>> 
>> If we migrate now, nothing stops us from adding new, causing someone else to 
>> be forced to migrate after…. Its one of the issues with the current enum, 
>> once it was created authors didn’t add to it always causing patches like 
>> this to try to migrate…
>> 
>> I am not trying to block your patch, if you don’t deal with this I am +0… 
>> just saying that maintaince is something we must think about
>> 
>> On Feb 9, 2023, at 6:37 AM, Miklosovic, Stefan 
>>  wrote:
>> 
>> Hi Maxim,
>> 
>> I would like to describe the ideal state / end goal, from my perspective.
>> 
>> All properties meant to be used only for tests would have a prefix like 
>> "cassandra.test.name.of.property" and production properties would be 
>> "cassandra.xyx". Once this is done, we can filter them out in vtable so 
>> there would not be any test-related properties in production. Test 
>> properties should be visible only when developing / testing Cassandra, in my 
>> opinion.
>> 
>> All other system properties should also have some consistent naming in place.
>> 
>> I understand that there is a lot of legacy in place and we can not rename 
>> properties just like that for people.
>> 
>> The approach I like is what was done to properties in cassandra.yaml. There 
>> is @Replaces annotation put on properties in Config which enables users to 
>> still use the old names.
>> 
>> I can imagine that something like this would used here. If an old name is 
>> specified, it would internally translate to a new name and only new names 
>> would be returned by vtable. There might be also a column for old names so 
>> people would know what new property the old one translates to and we should 
>> also emit warning for users that the system properties they are using are in 
>> the old format and they should move to the new ones.
>> 
>> Anyway, I am glad this is happening and we are making progress. It will be 
>> also way easier to dump all properties to the website when everything is 
>> centralized at once place.
>> 
>> Regards
>> 
>> 
>> From: Maxim Muzafarov 
>> Sent: Wednesday, February 8, 2023 19:48
>> To: dev@cassandra.apache.org
>> Subject: [DISCUSS] Moving system property names to the 
>> CassandraRelevantProperties
>> 
>> NetApp Security WARNING: This is an external email. Do not click links or 
>> open attachments unless you recognize the sender and know the content is 
>> safe.
>> 
>> 
>> 
>> 
>> Hello everyone,
>> 
>> 
>> We are trying to clean up the source code ar

Re: [DISCUSS] Moving system property names to the CassandraRelevantProperties

[David Capwell]

> All properties meant to be used only for tests would have a prefix like 
> "cassandra.test.name.of.property" and production properties would be 
> "cassandra.xyx". Once this is done, we can filter them out in vtable so there 
> would not be any test-related properties in production. Test properties 
> should be visible only when developing / testing Cassandra, in my opinion.
> 

Good point, I wouldn’t want to expose properties meant to break C* for testing… 
that implies to users we should be using it!

> I understand that there is a lot of legacy in place and we can not rename 
> properties just like that for people.

We could always look to do things like we did in Config (which you call out), 
add a way to “migrate” to the new naming/format.  I am not sure if there is 
enough configs to justify this, but with 5.0 happening it may be a good time to 
think about normalizing these.

> We are trying to clean up the source code around the direct use of system 
> properties and make this use more manageable and transparent.


I didn’t review, but one question I have is inline with "I would like to 
describe the ideal state / end goal”, but from the point of view of maintaining 
things…. If someone adds a new system property “should” they use this enum?  
“Should” we detect this was violated and fail the build?  

If we migrate now, nothing stops us from adding new, causing someone else to be 
forced to migrate after…. Its one of the issues with the current enum, once it 
was created authors didn’t add to it always causing patches like this to try to 
migrate…

I am not trying to block your patch, if you don’t deal with this I am +0… just 
saying that maintaince is something we must think about

> On Feb 9, 2023, at 6:37 AM, Miklosovic, Stefan  
> wrote:
> 
> Hi Maxim,
> 
> I would like to describe the ideal state / end goal, from my perspective.
> 
> All properties meant to be used only for tests would have a prefix like 
> "cassandra.test.name.of.property" and production properties would be 
> "cassandra.xyx". Once this is done, we can filter them out in vtable so there 
> would not be any test-related properties in production. Test properties 
> should be visible only when developing / testing Cassandra, in my opinion.
> 
> All other system properties should also have some consistent naming in place.
> 
> I understand that there is a lot of legacy in place and we can not rename 
> properties just like that for people.
> 
> The approach I like is what was done to properties in cassandra.yaml. There 
> is @Replaces annotation put on properties in Config which enables users to 
> still use the old names.
> 
> I can imagine that something like this would used here. If an old name is 
> specified, it would internally translate to a new name and only new names 
> would be returned by vtable. There might be also a column for old names so 
> people would know what new property the old one translates to and we should 
> also emit warning for users that the system properties they are using are in 
> the old format and they should move to the new ones.
> 
> Anyway, I am glad this is happening and we are making progress. It will be 
> also way easier to dump all properties to the website when everything is 
> centralized at once place.
> 
> Regards
> 
> 
> From: Maxim Muzafarov 
> Sent: Wednesday, February 8, 2023 19:48
> To: dev@cassandra.apache.org
> Subject: [DISCUSS] Moving system property names to the 
> CassandraRelevantProperties
> 
> NetApp Security WARNING: This is an external email. Do not click links or 
> open attachments unless you recognize the sender and know the content is safe.
> 
> 
> 
> 
> Hello everyone,
> 
> 
> We are trying to clean up the source code around the direct use of
> system properties and make this use more manageable and transparent.
> To achieve this, I have prepared a patch that moves all system
> property names to the CassandraRelevantProperties, which in turn makes
> some of the properties visible to a user through the
> SystemPropertiesTable virtual table.
> 
> The patch has passed a few rounds of review, but we still need another
> pair of eyes to make sure we are not missing anything valuable.
> Please, take a look at the patch.
> 
> You can find all the changes here:
> https://issues.apache.org/jira/browse/CASSANDRA-17797
> 
> 
> I'd also like to share the names of the properties that will appear in
> the SystemPropertiesTable, the appearance of which is related to the
> public API changes we agreed to discuss on the dev list.
> 
> 
> The public API changes
> 
> Newly production system properties added:
> 
> io.netty.eventLoopThreads
> io.netty.transport.estimateSizeOnSubmit
> java.security.auth.login.config
> javax.rmi.ssl.client.enabledCipherSuites
> javax.rmi.ssl.client.enabledProtocols
> ssl.enable
> log4j2.disable.jmx
> log4j2.shutdownHookEnabled
> logback.configurationFile
> 
> Newly added and used for tests only:
> 
> 

Re: [VOTE] CEP-21 Transactional Cluster Metadata

[David Capwell]

+1

> On Feb 7, 2023, at 7:15 AM, Jeremiah D Jordan  
> wrote:
> 
> +1 nb
> 
>> On Feb 6, 2023, at 10:15 AM, Sam Tunnicliffe  wrote:
>> 
>> Hi everyone,
>> 
>> I would like to start a vote on this CEP.
>> 
>> Proposal:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata
>> 
>> Discussion:
>> https://lists.apache.org/thread/h25skwkbdztz9hj2pxtgh39rnjfzckk7
>> 
>> The vote will be open for 72 hours.
>> A vote passes if there are at least three binding +1s and no binding vetoes.
>> 
>> Thanks,
>> Sam
> 

Re: Implicitly enabling ALLOW FILTERING on virtual tables

[David Capwell]

> I don't think the assumption that "virtual tables will always be small and 
> always fit in memory" is a safe one.

Agree, there is a repair ticket to have the coordinating node do network 
queries to peers to resolve the table (rather than operator querying 
everything, allow the coordinator node to do it for you)… so this assumption 
may not be true down the line.

I could be open to a table property that says ALLOW FILTERING on by default or 
not… then we can pick and choose vtables (or have vtables opt-out)…. I kinda 
like like the lack of consistency with this approach though

> On Feb 3, 2023, at 11:24 AM, C. Scott Andreas  wrote:
> 
> There are some ideas that development community members have kicked around 
> that may falsify the assumption that "virtual tables are tiny and will fit in 
> memory."
> 
> One example is CASSANDRA-14629: Abstract Virtual Table for very large result 
> sets
> https://issues.apache.org/jira/browse/CASSANDRA-14629
> 
> Chris's proposal here is to enable query results from virtual tables to be 
> streamed to the client rather than being fully materialized. There are some 
> neat possibilities suggested in this ticket, such as debug functionality to 
> dump the contents of a raw SSTable via the CQL interface, or the contents of 
> the database's internal caches. One could also imagine a feature like this 
> providing functionality similar to a foreign data wrapper in other databases.
> 
> I don't think the assumption that "virtual tables will always be small and 
> always fit in memory" is a safe one.
> 
> I don't think we should implicitly add "ALLOW FILTERING" to all queries 
> against virtual tables because of this, in addition to concern with departing 
> from standard CQL semantics for a type of tables deemed special.
> 
> – Scott
> 
>> On Feb 3, 2023, at 6:52 AM, Maxim Muzafarov  wrote:
>> 
>> 
>> Hello Stefan,
>> 
>> Regarding the decision to implicitly enable ALLOW FILTERING for
>> virtual tables, which also makes sense to me, it may be necessary to
>> consider changing the clustering columns in the virtual table metadata
>> to regular columns as well. The reasons are the same as mentioned
>> earlier: the virtual tables hold their data in memory, thus we do not
>> benefit from the advantages of ordered data (e.g. the ClientsTable and
>> its ClusteringColumn(PORT)).
>> 
>> Changing the clustering column to a regular column may simplify the
>> virtual table data model, but I'm afraid it may affect users who rely
>> on the table metadata.
>> 
>> 
>> 
>> On Fri, 3 Feb 2023 at 12:32, Andrés de la Peña  wrote:
>>> 
>>> I think removing the need for ALLOW FILTERING on virtual tables makes sense 
>>> and would be quite useful for operators.
>>> 
>>> That guard exists for performance issues that shouldn't occur on virtual 
>>> tables. We also have a flag in case some future virtual table 
>>> implementation has limitations regarding filtering, although it seems it's 
>>> not the case with any of the existing virtual tables.
>>> 
>>> It is not like we would promote bad habits because virtual tables are meant 
>>> to be queried by operators / administrators only.
>>> 
>>> 
>>> It might even be quite the opposite, since in the current situation users 
>>> might get used to routinely use ALLOW FILTERING for querying their virtual 
>>> tables.
>>> 
>>> It has been mentioned on the #cassandra-dev Slack thread where this started 
>>> (1) that it's kind of an API inconsistency to allow querying by non-primary 
>>> keys on virtual tables without ALLOW FILTERING, whereas it's required for 
>>> regular tables. I think that a simply doc update saying that virtual 
>>> tables, which are not regular tables, support filtering would be enough. 
>>> Virtual tables are well identified by both the keyspace they belong to and 
>>> doc, so users shouldn't have trouble knowing whether a table is virtual. It 
>>> would be similar to the current exception for ALLOW FILTERING, where one 
>>> needs to use it unless the table has an index for the queried column.
>>> 
>>> (1) https://the-asf.slack.com/archives/CK23JSY2K/p1675352759267329
>>> 
>>> On Fri, 3 Feb 2023 at 09:09, Miklosovic, Stefan 
>>>  wrote:
 
 Hi list,
 
 the content of virtual tables is held in memory (and / or is fetched every 
 time upon request). While doing queries against such table for a column 
 outside of primary key, normally, users are required to specify ALLOW 
 FILTERING. This makes total sense for "ordinary tables" for applications 
 to have performant and effective queries but it kinds of loses the 
 applicability for virtual tables when it literally holds just handful of 
 entries in memory and it just does not matter, does it?
 
 What do you think about implicitly allowing filtering for virtual tables 
 so we save ourselves from these pesky errors when we want to query 
 arbitrary column and we need to satisfy CQL spec just to do that?
 
 It is not like 

Re: Welcome Patrick McFadin as Cassandra Committer

[David Capwell]

Congrats and welcome! =)

> On Feb 2, 2023, at 10:53 AM, J. D. Jordan  wrote:
> 
> Congrats!
> 
>> On Feb 2, 2023, at 12:47 PM, Christopher Bradford  
>> wrote:
>> 
>> 
>> Congrats Patrick! Well done. 
>> 
>> On Thu, Feb 2, 2023 at 10:44 AM Aaron Ploetz > > wrote:
>>> Patrick FTW!!!
>>> 
>>> On Thu, Feb 2, 2023 at 12:32 PM Joseph Lynch >> > wrote:
 W! Congratulations Patrick!!
 
 -Joey
 
 On Thu, Feb 2, 2023 at 9:58 AM Benjamin Lerer >>> > wrote:
> The PMC members are pleased to announce that Patrick McFadin has accepted
> the invitation to become committer today.
> 
> Thanks a lot, Patrick, for everything you have done for this project and 
> its community through the years.
> 
> Congratulations and welcome!
> 
> The Apache Cassandra PMC members
>> -- 
>> 
>> Christopher Bradford
>> 

Re: Merging CEP-15 to trunk

[David Capwell]

> It's been mentioned "always shippable trunk according to circleci".  That's 
> not true: we are always shippable according to *either* CI.  There are folk 
> just using ci-cassandra for our pre-commit gateway.  It is important that you 
> don't trash the other CI system, particularly when it comes to parity of the 
> tests that we run.  If you introduce new tests in one CI it is on you to 
> introduce it to the other CI, and therefore run both CI pre-merge.  This does 
> not imply a merge is blocked on green ci-cassandra.a.o

I had no intention to “trash” Jenkins, my reaction was to the statements that a 
lack of Jenkins means we are not following the process.  I am fully aware that 
there is a lot of work here, and know many of us (including myself) want to 
migrate away from Circle Ci.


> On Feb 1, 2023, at 3:24 PM, Mick Semb Wever  wrote:
> 
> 
> 
>> Hi Everyone, I hope you all had a lovely holiday period.
> 
> 
> Thank you David and Caleb for your continuous work dealing with the 
> practicalities involved in getting the merge to happen!  And thank you 
> Benedict for starting the thread – it is an important topic for us to 
> continue as CEPs and feature branches become a thing!
> 
> Reading the thread I see some quite distinct perspectives, and it's making it 
> difficult to have as constructive a dialogue as we should wish for.  As is 
> usual we see this balanced by the  productive collaboration happening but 
> hidden in the jira tickets.
> 
> I want to raise the following points, not as things to solve necessarily but 
> to help us appreciate just how different our PoVs are and how easy it is for 
> us to be talking past each other. If you immediately disagree with any of 
> them I suggest you read below how my PoV gets me there.
> 
> 1. Work based on a notion of meritocracy. If you do the work, or have done 
> the work, your input weighs more.
> 2. Promote teamwork and value an inclusive culture.
> 3. Where the work is, or has been committed to, can be considered irrelevant 
> (in the context of this thread).
> 4. There are different types of reviews. We want to embrace them all.
> 5. We gate on either CI (not only circleci). Do not trash or reduce parity in 
> the other.
> 6. A feature branch != release branch/trunk. 
> 7. The cep-15-accord was not ready to merge, this thread helped smoke out a 
> number of minor things.
> 8. Merging of feature branches should be quick and light team-work.
> 
> 
> Going into these in more detail…
> 
> 1+3) 
> If a patch from a pmc engineer has been reviewed by two other pmc engineers,  
> CI has been run,  and they are about to merge,  a new reviewer interjecting 
> late can expect very little to happen pre-merge.  If there's something 
> serious it needs to be explained quickly.  We value people's time, 
> productiveness, and expertise.
> 
> This applies everywhere in the project: from tickets with patch files to CEP 
> work and feature branches. 
> 
> 2)
> The more eyes the better.  Everything we do is transparent, and we should 
> always be receptive and by default open to new input regardless of where in 
> the process we are and where it comes from.  First be receptive and hear 
> people out, afterwards make the "now, or later" decision.  Take the time to 
> explain why a "we'll do it later" decision – it might not be obvious to 
> everyone, it helps newcomers understand the project dynamics, and it sets 
> project precedence.  Use (1) as needed. 
> 
> 4)
> A lot of the thread has focused on reviews of Accord and its technical 
> components.  There are other types of reviews.  Examples are docs (internal 
> and external), the build system, our CIs and integrations, releases and ASF 
> policies.  Henrik pointed out that it is wasteful for these reviews to happen 
> early in the review window.  In addition, they are quite small in comparison 
> to initial technical reviews and equally quick to address. The larger the 
> patch the more we can expect others stepping in with these types of tail-end 
> reviews.
> 
> If we think of the review process as promoting teamwork and an inclusive 
> culture, we can also think of reviews akin to pair-programming that help 
> mentor those less knowledgeable in an area of the code.  This needs to be 
> time-boxed ofc, but IMHO we should keep an open mind to it – these types of 
> reviews do surprisingly lead to bug fixes and improvements with the reviewer 
> acting as a rubber-duck. This type of review we would want early, not late.
> 
> There's also another type of review here.  If the author of another CEP has 
> another feature branch about to merge, they may want to review the branch to 
> get ahead of possible incoming conflicts.  This is a review that cares 
> nothing about Accord itself.  Their evaluation is if there's any project-wide 
> ROI if some changes happen before the merge.  Again, teamwork.
> 
> 5)
> It's been mentioned "always shippable trunk according to circleci".  That's 
> not true: we are always 

Re: Merging CEP-15 to trunk

[David Capwell]

> About this merge - I would  personally love to see everything rebased and 
> things rerun in CI which no one has a doubt it will happen

Yes, I think this is expected behavior that must happen before adding to trunk, 
or even just rebasing the feature branch; CI must be passing and flaky tests 
(as you called out) must be resolved.  Nothing is special from a process point 
of view as this is already the expected process.  I know Caleb is working on it 
and uses my circleci script, which leverages the _repeat job to help flesh this 
out.  With the merge to trunk it “should” detect all new tests and run them, so 
its fair to expect this to be stable.

> I am not sure I understand this - 


Sorry, I was trying to be explicit on the current state of test failures, but 
was not clear.

Accord adds a new system table to make state durable (similar to the paxos 
table).  This added table causes a well known subset of python dtests to fail 
as they are checking for the set of keyspaces/tables and detecting a new one 
added, causing the test to fail.  We have been ignoring these rather than 
changing python-dtest that way we pull in latest changes there rather than 
forking and freezing the branch we use.  We will 100% update python dtest along 
with the merge to trunk similar to all other cross-cutting work done.  Trunk 
must not fail due to this table getting added and must 100% be dealt with at 
the same time we merge.

> Do you plan to merge before fixing the tests? I am confused


Yes, its a blocker for trunk merge.  

> I think Henrik brought an interesting point about releasable trunk. I don’t 
> think that we are yet there with the Jenkins issues, but we are getting close 
> and we strive for that, no?


As Josh calls out, we require releasable trunk as defined by Circle CI (change 
was made in 4.1); my understanding is that people are working to improve 
Jenkins, and once this work is done we can migrate off Circle CI.  Until that 
happens we need to make sure Circle CI is stable for every commit.

> 3) what is the story with the ninja fixes? 


Can’t speak for 100% of them, but from the ones I have seen it boils down to 
“our merge process relies on humans to do the right thing, and people mess up 
some times”, which sadly is true in trunk.  To avoid calling anyone out other 
than myself, lets look at a specific one that I did

$ git show f8243f41c9e96c4a0390558086ece078b0b6b15c
commit f8243f41c9e96c4a0390558086ece078b0b6b15c
Author: David Capwell 
Date:   Mon Jan 9 13:20:58 2023 -0800

Ninja: Add AccordTestUtils.parse which was missing in the latest commit

diff --git a/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java 
b/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
index 4adad32d8a..20142c439b 100644
--- a/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
+++ b/test/unit/org/apache/cassandra/service/accord/AccordTestUtils.java
@@ -174,6 +174,14 @@ public class AccordTestUtils
 return statement.createTxn(ClientState.forInternalCalls(), options);
 }

+public static TransactionStatement parse(String query)
+{
+TransactionStatement.Parsed parsed = (TransactionStatement.Parsed) 
QueryProcessor.parseStatement(query);
+Assert.assertNotNull(parsed);
+TransactionStatement statement = (TransactionStatement) 
parsed.prepare(ClientState.forInternalCalls());
+return statement;
+}
+
 public static Txn createTxn(int readKey, int... writeKeys)
 {
 StringBuilder sb = new StringBuilder("BEGIN TRANSACTION\n”);

I merged but a specific method was missing, this caused the branch to fail to 
compile so I ninja patched by adding that method back; this was reviewed in 
slack with Benedict (if I remember correctly).

I made a human judgement to merge via GH UI and bypass my merge scripts, as I 
had a Circle CI build showing success… I messed up and had to go back and fix 
it after commit.

> It is frustrating and I can imagine how after that much work around Accord 
> people just want it in


I do not believe anyone is arguing to merge Accord with bugs… bugs we know 
about are blockers for merge.  There are pending improvements and features 
people are working on, but I don’t believe I know of us trying to merge bugs 
into trunk.  If there are bugs found we should address before merge, but 
improvements can happen after trunk merge (depending on timing, before merge is 
fine).

I do hope this comment doesn’t cause a rabbit hole of “what is a bug vs what is 
an improvement”….


> On Jan 31, 2023, at 8:20 AM, Josh McKenzie  wrote:
> 
>> Don't we follow a principle of always shippable trunk? This was actually a 
>> reason why I sidelined the talk about post-merge review, because it implies 
>> that the code wasn't "good enough"/perfect when it was first merged.
> We follow a principle of "always shippable trunk accord

Re: Merging CEP-15 to trunk

[David Capwell]

> But during development, did you ever run nightly tests / all tests?


I strongly recommend people I work with to use my script that modifies circle 
ci, which ops us into running all tests; I tend to see people use this as it 
makes CI faster, but has this by product as well.

So yes, all tests should be run before merge.  There are examples of Jenkins 
only tests that are not run, but again this is due to existing limitations with 
Jenkins.

> On Jan 30, 2023, at 3:33 PM, Henrik Ingo  wrote:
> 
> On Tue, Jan 31, 2023 at 1:28 AM David Capwell  <mailto:dcapw...@apple.com>> wrote:
>> If the CI coverage isn't 100%, then we should just identify the gaps, and 
>> focus on that while preparing to merge
> 
> It has 100% coverage that is done normally for trunk merges; which is a 
> pre-commit CI run in Circle OR Jenkins.
> 
> 
> Sure. And thanks.
> 
> But during development, did you ever run nightly tests / all tests? I 
> wouldn't want the night after merging to trunk to be the first time those are 
> run.
> 
> henrik
> -- 
> 
> Henrik Ingo
> c. +358 40 569 7354 
> w. www.datastax.com <http://www.datastax.com/>
>  <https://www.facebook.com/datastax>   <https://twitter.com/datastax>   
> <https://www.linkedin.com/company/datastax/>   <https://github.com/datastax/>