Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[German Eichberger via dev]

+1

From: Patrick McFadin 
Sent: Friday, December 22, 2023 9:12 AM
To: dev@cassandra.apache.org 
Subject: [EXTERNAL] Re: Harry in-tree (Forked from "Long tests, Burn tests, 
Simulator tests, Fuzz tests - can we clarify the diffs?")

It was great having some more extended discussions about Harry in person last 
week. Anything we can do to make it easier for anyone to test Cassandra 
thoroughly is an easy +1 from me!

Thanks for all your efforts so far, Alex.

Patrick

On Fri, Dec 22, 2023 at 8:03 AM Jacek Lewandowski 
mailto:lewandowski.ja...@gmail.com>> wrote:
Obviously +1

Thank you Alex

pt., 22 gru 2023, 16:45 użytkownik Sumanth Pasupuleti 
mailto:sumanth.pasupuleti...@gmail.com>> 
napisał:
+1, thank you for your efforts in bringing Harry in-tree. Anything that 
improves the testing ecosystem for Cassandra, particularly around complex 
scenarios / edge cases  goes a long way in improving reliability, and with 
having a powerful tool like Harry in-tree, it is a lot more accessible to the 
developers than it has been. Also, thank you for keeping in mind the onboarding 
experience of developers.

- Sumanth

On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov 
mailto:al...@coffeenco.de>> wrote:
Some follow-up tickets to establish the project direction:

https://issues.apache.org/jira/browse/CASSANDRA-19229

Two other things that we will work on in Tree are:
https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM test 
for partition-restricted 2i queries)
https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded SAI read 
and write fuzz test)

If you would like to get your recently added feature tested with Harry model, 
please let me know!

On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
+1

Sounds like a great change that will help us unify around a common testing 
paradigm, and even pave the path to in-tree load testing plus integrated 
correctness checking which would be extremely valuable!

-Joey

On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe 
mailto:calebrackli...@gmail.com>> wrote:
+1

Agree w/ all the justifications mentioned above.

As a reviewer on 
CASSANDRA-19210, my 
goals were to a.) look at the directory, naming, and package structure of the 
ported code, b.) make sure IDE integration was working, and c.) make sure any 
modifications to existing code (rather than direct code movements from 
cassandra-harry) were straightforward.

On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov 
mailto:al...@coffeenco.de>> wrote:

Hey folks,

I am mostly done with a patch that brings Harry in-tree [1]. I will trigger one 
more CI run overnight, and my intention was to merge it some time soon, but I 
wanted to give a fair warning here, since this is a relatively large patch.

Good news for everyone that it:
  a) touches no production code whatsoever. Only test (in-jvm dtest namely) 
code that was using Harry already.
  b) the only tests that are changed are ones that used a duplicate version of 
placement simulator we had both for testing TCM, and in Harry
  c) in addition, I have converted 3 existing TCM tests to a new API to have 
some base for examples/usage.

Since we were effectively relying on this code for a while now, and the 
intention now is to converge to:
  a) fewer different generators, and have a shareable version of generators for 
everyone to use accross the base
  b) a testing tool that can be useful for both trivial cases, and complex 
scenarios
myself and many other Cassandra contributors have expressed an opinion that 
bringing Harry in-tree will be highly benefitial.

I strongly believe that bringing Harry in-tree will help to lower the barrier 
for fuzz test and simplify co-development of Cassandra and Harry. Previously, 
it has been rather difficult to debug edge cases because I had to either 
re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry jar 
and bring it to Cassandra, which is both tedious and time consuming. Moreover, 
I believe we have missed at very least one RT regression [2] because Harry was 
not in-tree, as its tests would've caught the issue even with the model that 
existed.

For other recently found issues, I think having Harry in-tree would have 
substantially lowered a turnaround time, and allowed me to share repros with 
developers of corresponding features much quicker.

I do expect a slight learning curve for Harry, but my intention is to build a 
web of simple tests (worked on some of them yesterday after conversation with 
David already), which can follow the in-jvm-dtest pattern of find-similar-test 
/ copy / modify. There's already copious documentation, so I do not believe not 
having docs for Harry was ever an issue, since there have been plenty.

You all are aware of my dedication to testing and quality of Apache Cassandra, 
and I hope you also see the benefits of having a model checker in-tree.

Thank you and happy upcoming h

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Patrick McFadin]

It was great having some more extended discussions about Harry in person
last week. Anything we can do to make it easier for anyone to test
Cassandra thoroughly is an easy +1 from me!

Thanks for all your efforts so far, Alex.

Patrick

On Fri, Dec 22, 2023 at 8:03 AM Jacek Lewandowski <
lewandowski.ja...@gmail.com> wrote:

> Obviously +1
>
> Thank you Alex
>
> pt., 22 gru 2023, 16:45 użytkownik Sumanth Pasupuleti <
> sumanth.pasupuleti...@gmail.com> napisał:
>
>> +1, thank you for your efforts in bringing Harry in-tree. Anything that
>> improves the testing ecosystem for Cassandra, particularly around complex
>> scenarios / edge cases  goes a long way in improving reliability, and with
>> having a powerful tool like Harry in-tree, it is a lot more accessible to
>> the developers than it has been. Also, thank you for keeping in mind the
>> onboarding experience of developers.
>>
>> - Sumanth
>>
>> On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov  wrote:
>>
>>> Some follow-up tickets to establish the project direction:
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-19229
>>>
>>> Two other things that we will work on in Tree are:
>>> https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM
>>> test for partition-restricted 2i queries)
>>> https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded
>>> SAI read and write fuzz test)
>>>
>>> If you would like to get your recently added feature tested with Harry
>>> model, please let me know!
>>>
>>> On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
>>>
>>> +1
>>>
>>> Sounds like a great change that will help us unify around a common
>>> testing paradigm, and even pave the path to in-tree load testing plus
>>> integrated correctness checking which would be extremely valuable!
>>>
>>> -Joey
>>>
>>> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe <
>>> calebrackli...@gmail.com> wrote:
>>>
>>> +1
>>>
>>> Agree w/ all the justifications mentioned above.
>>>
>>> As a reviewer on CASSANDRA-19210
>>> , my goals were
>>> to a.) look at the directory, naming, and package structure of the ported
>>> code, b.) make sure IDE integration was working, and c.) make sure any
>>> modifications to existing code (rather than direct code movements from
>>> cassandra-harry) were straightforward.
>>>
>>> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:
>>>
>>>
>>> Hey folks,
>>>
>>> I am mostly done with a patch that brings Harry in-tree [1]. I will
>>> trigger one more CI run overnight, and my intention was to merge it some
>>> time soon, but I wanted to give a fair warning here, since this is a
>>> relatively large patch.
>>>
>>> Good news for everyone that it:
>>>   a) touches no production code whatsoever. Only test (in-jvm dtest
>>> namely) code that was using Harry already.
>>>   b) the only tests that are changed are ones that used a duplicate
>>> version of placement simulator we had both for testing TCM, and in Harry
>>>   c) in addition, I have converted 3 existing TCM tests to a new API to
>>> have some base for examples/usage.
>>>
>>> Since we were effectively relying on this code for a while now, and the
>>> intention now is to converge to:
>>>   a) fewer different generators, and have a shareable version of
>>> generators for everyone to use accross the base
>>>   b) a testing tool that can be useful for both trivial cases, and
>>> complex scenarios
>>> myself and many other Cassandra contributors have expressed an opinion
>>> that bringing Harry in-tree will be highly benefitial.
>>>
>>> I strongly believe that bringing Harry in-tree will help to lower the
>>> barrier for fuzz test and simplify co-development of Cassandra and Harry.
>>> Previously, it has been rather difficult to debug edge cases because I had
>>> to either re-compile an in-jvm dtest jar and bring it to Harry, or
>>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
>>> time consuming. Moreover, I believe we have missed at very least one RT
>>> regression [2] because Harry was not in-tree, as its tests would've caught
>>> the issue even with the model that existed.
>>>
>>> For other recently found issues, I think having Harry in-tree would have
>>> substantially lowered a turnaround time, and allowed me to share repros
>>> with developers of corresponding features much quicker.
>>>
>>> I do expect a slight learning curve for Harry, but my intention is to
>>> build a web of simple tests (worked on some of them yesterday after
>>> conversation with David already), which can follow the in-jvm-dtest pattern
>>> of find-similar-test / copy / modify. There's already copious
>>> documentation, so I do not believe not having docs for Harry was ever an
>>> issue, since there have been plenty.
>>>
>>> You all are aware of my dedication to testing and quality of Apache
>>> Cassandra, and I hope you also see the benefits of having a model checker
>>> in-tree.
>>>
>>> Thank you and happy upcoming ho

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Francisco Guerrero]

+1 thanks for this effort! 

On 2023/12/21 21:22:54 Alex Petrov wrote:
> Hey folks,
> 
> I am mostly done with a patch that brings Harry in-tree [1]. I will trigger 
> one more CI run overnight, and my intention was to merge it some time soon, 
> but I wanted to give a fair warning here, since this is a relatively large 
> patch. 
> 
> Good news for everyone that it: 
>   a) touches no production code whatsoever. Only test (in-jvm dtest namely) 
> code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate version 
> of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to have 
> some base for examples/usage.
> 
> Since we were effectively relying on this code for a while now, and the 
> intention now is to converge to: 
>   a) fewer different generators, and have a shareable version of generators 
> for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex 
> scenarios 
> myself and many other Cassandra contributors have expressed an opinion that 
> bringing Harry in-tree will be highly benefitial.
> 
> I strongly believe that bringing Harry in-tree will help to lower the barrier 
> for fuzz test and simplify co-development of Cassandra and Harry. Previously, 
> it has been rather difficult to debug edge cases because I had to either 
> re-compile an in-jvm dtest jar and bring it to Harry, or re-compile a Harry 
> jar and bring it to Cassandra, which is both tedious and time consuming. 
> Moreover, I believe we have missed at very least one RT regression [2] 
> because Harry was not in-tree, as its tests would've caught the issue even 
> with the model that existed.
> 
> For other recently found issues, I think having Harry in-tree would have 
> substantially lowered a turnaround time, and allowed me to share repros with 
> developers of corresponding features much quicker.
> 
> I do expect a slight learning curve for Harry, but my intention is to build a 
> web of simple tests (worked on some of them yesterday after conversation with 
> David already), which can follow the in-jvm-dtest pattern of 
> find-similar-test / copy / modify. There's already copious documentation, so 
> I do not believe not having docs for Harry was ever an issue, since there 
> have been plenty. 
> 
> You all are aware of my dedication to testing and quality of Apache 
> Cassandra, and I hope you also see the benefits of having a model checker 
> in-tree.
> 
> Thank you and happy upcoming holidays,
> --Alex
> 
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
> 

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Jacek Lewandowski]

Obviously +1

Thank you Alex

pt., 22 gru 2023, 16:45 użytkownik Sumanth Pasupuleti <
sumanth.pasupuleti...@gmail.com> napisał:

> +1, thank you for your efforts in bringing Harry in-tree. Anything that
> improves the testing ecosystem for Cassandra, particularly around complex
> scenarios / edge cases  goes a long way in improving reliability, and with
> having a powerful tool like Harry in-tree, it is a lot more accessible to
> the developers than it has been. Also, thank you for keeping in mind the
> onboarding experience of developers.
>
> - Sumanth
>
> On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov  wrote:
>
>> Some follow-up tickets to establish the project direction:
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-19229
>>
>> Two other things that we will work on in Tree are:
>> https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM
>> test for partition-restricted 2i queries)
>> https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded
>> SAI read and write fuzz test)
>>
>> If you would like to get your recently added feature tested with Harry
>> model, please let me know!
>>
>> On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
>>
>> +1
>>
>> Sounds like a great change that will help us unify around a common
>> testing paradigm, and even pave the path to in-tree load testing plus
>> integrated correctness checking which would be extremely valuable!
>>
>> -Joey
>>
>> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe 
>> wrote:
>>
>> +1
>>
>> Agree w/ all the justifications mentioned above.
>>
>> As a reviewer on CASSANDRA-19210
>> , my goals were
>> to a.) look at the directory, naming, and package structure of the ported
>> code, b.) make sure IDE integration was working, and c.) make sure any
>> modifications to existing code (rather than direct code movements from
>> cassandra-harry) were straightforward.
>>
>> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:
>>
>>
>> Hey folks,
>>
>> I am mostly done with a patch that brings Harry in-tree [1]. I will
>> trigger one more CI run overnight, and my intention was to merge it some
>> time soon, but I wanted to give a fair warning here, since this is a
>> relatively large patch.
>>
>> Good news for everyone that it:
>>   a) touches no production code whatsoever. Only test (in-jvm dtest
>> namely) code that was using Harry already.
>>   b) the only tests that are changed are ones that used a duplicate
>> version of placement simulator we had both for testing TCM, and in Harry
>>   c) in addition, I have converted 3 existing TCM tests to a new API to
>> have some base for examples/usage.
>>
>> Since we were effectively relying on this code for a while now, and the
>> intention now is to converge to:
>>   a) fewer different generators, and have a shareable version of
>> generators for everyone to use accross the base
>>   b) a testing tool that can be useful for both trivial cases, and
>> complex scenarios
>> myself and many other Cassandra contributors have expressed an opinion
>> that bringing Harry in-tree will be highly benefitial.
>>
>> I strongly believe that bringing Harry in-tree will help to lower the
>> barrier for fuzz test and simplify co-development of Cassandra and Harry.
>> Previously, it has been rather difficult to debug edge cases because I had
>> to either re-compile an in-jvm dtest jar and bring it to Harry, or
>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
>> time consuming. Moreover, I believe we have missed at very least one RT
>> regression [2] because Harry was not in-tree, as its tests would've caught
>> the issue even with the model that existed.
>>
>> For other recently found issues, I think having Harry in-tree would have
>> substantially lowered a turnaround time, and allowed me to share repros
>> with developers of corresponding features much quicker.
>>
>> I do expect a slight learning curve for Harry, but my intention is to
>> build a web of simple tests (worked on some of them yesterday after
>> conversation with David already), which can follow the in-jvm-dtest pattern
>> of find-similar-test / copy / modify. There's already copious
>> documentation, so I do not believe not having docs for Harry was ever an
>> issue, since there have been plenty.
>>
>> You all are aware of my dedication to testing and quality of Apache
>> Cassandra, and I hope you also see the benefits of having a model checker
>> in-tree.
>>
>> Thank you and happy upcoming holidays,
>> --Alex
>>
>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>
>>
>>

Meet our keynote speakers and register to Community Over Code EU!

[Ryan Skraba]

[Note: You're receiving this email because you are subscribed to one or
more project dev@ mailing lists at the Apache Software Foundation.]











*
Merge
with the ASF EUniverse!The registration for Community Over Code Europe is
finally open! Get your tickets now and save your spot!
We are happy to announce that we
have confirmed the first featured speakers
!  - Asim Hussain, Executive
Director at Green Software Foundation- Dirk-Willem Van Gulik, VP of Public
Policy at The Apache Software Foundation- Ruth Ikega, Community Lead at
CHAOSS Africa Visit our website
 to learn more about this
amazing lineup.CFP is openWe are looking forward to hearing all you have to
share with the Apache Community. Please submit your talk proposal
 before January 12, 2024.Interested in
boosting your brand?Take a look at our prospectus

and find out the opportunities we have for you. Be one step ahead and book
your room at the hotel venueWe have a special rate for you at the Radisson
Blu Carlton, the hotel that will hold Community Over Code EU. Learn more
about the location and venue 
and book your accommodation. Should you have any questions, please do not
hesitate to contact us. We wish you Happy Holidays in the company of your
loved ones! See you in Bratislava next year!Community Over Code EU
Organizer Committee*

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Sumanth Pasupuleti]

+1, thank you for your efforts in bringing Harry in-tree. Anything that
improves the testing ecosystem for Cassandra, particularly around complex
scenarios / edge cases  goes a long way in improving reliability, and with
having a powerful tool like Harry in-tree, it is a lot more accessible to
the developers than it has been. Also, thank you for keeping in mind the
onboarding experience of developers.

- Sumanth

On Fri, Dec 22, 2023 at 1:11 AM Alex Petrov  wrote:

> Some follow-up tickets to establish the project direction:
>
> https://issues.apache.org/jira/browse/CASSANDRA-19229
>
> Two other things that we will work on in Tree are:
> https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM
> test for partition-restricted 2i queries)
> https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded SAI
> read and write fuzz test)
>
> If you would like to get your recently added feature tested with Harry
> model, please let me know!
>
> On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
>
> +1
>
> Sounds like a great change that will help us unify around a common testing
> paradigm, and even pave the path to in-tree load testing plus integrated
> correctness checking which would be extremely valuable!
>
> -Joey
>
> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe 
> wrote:
>
> +1
>
> Agree w/ all the justifications mentioned above.
>
> As a reviewer on CASSANDRA-19210
> , my goals were to
> a.) look at the directory, naming, and package structure of the ported
> code, b.) make sure IDE integration was working, and c.) make sure any
> modifications to existing code (rather than direct code movements from
> cassandra-harry) were straightforward.
>
> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:
>
>
> Hey folks,
>
> I am mostly done with a patch that brings Harry in-tree [1]. I will
> trigger one more CI run overnight, and my intention was to merge it some
> time soon, but I wanted to give a fair warning here, since this is a
> relatively large patch.
>
> Good news for everyone that it:
>   a) touches no production code whatsoever. Only test (in-jvm dtest
> namely) code that was using Harry already.
>   b) the only tests that are changed are ones that used a duplicate
> version of placement simulator we had both for testing TCM, and in Harry
>   c) in addition, I have converted 3 existing TCM tests to a new API to
> have some base for examples/usage.
>
> Since we were effectively relying on this code for a while now, and the
> intention now is to converge to:
>   a) fewer different generators, and have a shareable version of
> generators for everyone to use accross the base
>   b) a testing tool that can be useful for both trivial cases, and complex
> scenarios
> myself and many other Cassandra contributors have expressed an opinion
> that bringing Harry in-tree will be highly benefitial.
>
> I strongly believe that bringing Harry in-tree will help to lower the
> barrier for fuzz test and simplify co-development of Cassandra and Harry.
> Previously, it has been rather difficult to debug edge cases because I had
> to either re-compile an in-jvm dtest jar and bring it to Harry, or
> re-compile a Harry jar and bring it to Cassandra, which is both tedious and
> time consuming. Moreover, I believe we have missed at very least one RT
> regression [2] because Harry was not in-tree, as its tests would've caught
> the issue even with the model that existed.
>
> For other recently found issues, I think having Harry in-tree would have
> substantially lowered a turnaround time, and allowed me to share repros
> with developers of corresponding features much quicker.
>
> I do expect a slight learning curve for Harry, but my intention is to
> build a web of simple tests (worked on some of them yesterday after
> conversation with David already), which can follow the in-jvm-dtest pattern
> of find-similar-test / copy / modify. There's already copious
> documentation, so I do not believe not having docs for Harry was ever an
> issue, since there have been plenty.
>
> You all are aware of my dedication to testing and quality of Apache
> Cassandra, and I hope you also see the benefits of having a model checker
> in-tree.
>
> Thank you and happy upcoming holidays,
> --Alex
>
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>
>
>

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Josh McKenzie]

> I don't even think we have to think about *new* SAI features to see where it 
> will benefit from further *local* optimization...
You make good points IMO. After Caleb's reasoning it makes sense to me to start 
working on query optimization w/even just our initial SAI feature-set given 
querying across multiple indices.

On Fri, Dec 22, 2023, at 8:42 AM, J. D. Jordan wrote:
> 
> The CEP-29 “rejected alternatives” section mentions one such use case.  Being 
> able to put NOT arbitrarily in a query.  Adding an OR operator is another 
> thing we are likely to want to do in the near future that would benefit from 
> this work, those benefit from the syntax tree and reordering parts of the 
> proposal.
> 
> But I think we already have enough complexity available to us to justify a 
> query optimizer in the fact of multi index queries today. Especially when you 
> have the new ANN OF operator in use combined with index queries.  Depending 
> on what order you query the indexes in, it can dramatically change the 
> performance of the query.  We are seeing and working through such issues in 
> Astra today.
> 
> -Jeremiah
> 
> 
>> On Dec 21, 2023, at 12:00 PM, Josh McKenzie  wrote:
>> 
>>> we are already late. We have several features running in production that we 
>>> chose to not open source yet because implementing phase 1 of the CEP would 
>>> have heavily simplify their designs. The cost of developing them was much 
>>> higher than what it would have been if the CEP had already been 
>>> implemented. We are also currently working on some SAI features that need 
>>> cost based optimization.
>> Are there DISCUSS threads or CEP's for any of that work? For us to have a 
>> useful discussion about whether we're at a point in the project where a 
>> query optimizer is appropriate for the project this information would be 
>> vital.
>> 
>> On Thu, Dec 21, 2023, at 12:33 PM, Benjamin Lerer wrote:
>>> Hey German,
>>> 
>>> To clarify things, we intend to push cardinalities across nodes, not costs. 
>>> It will be up to the Cost Model to estimate cost based on those 
>>> cardinalities. We will implement some functionalities to collect costs on 
>>> query execution to be able to provide them as the output of EXPLAIN ANALYZE.
>>> 
>>> We will provide more details on how we will collect and distribute 
>>> cardinalities. We will probably not go into details on how we will estimate 
>>> costs before the patch for it is ready. The main reason being that there 
>>> are a lot of different parts that you need to account for and that it will 
>>> require significant testing and experimentation.
>>> 
>>> Regarding multi-tenancy, even if you use query cost, do not forget that you 
>>> will have to account also for background tasks such as compaction, repair, 
>>> backup, ... which is not included in this CEP.  
>>> 
>>> Le jeu. 21 déc. 2023 à 00:18, German Eichberger via dev 
>>>  a écrit :
 All,
 
 very much agree with Scott's reasoning. 
 
 It seems expedient given the advent of ACCORD transactions to be more like 
 the other distributed SQL databases and just support SQL. But just because 
 it's expedient it isn't right and we should work out the relational 
 features in more detail before we embark on tying us to some query 
 planning design.
 
 The main problem in this space is pushing cost / across nodes based on 
 data density. I understand that TCM will level out data density but the 
 cost based optimizer proposal does a lot of hand waving when it comes to 
 collecting/estimating costs for each node. I like to see more details on 
 this since otherwise it will be fairly limiting.
 
 I am less tied to ALLOW FILTERING - many of my customers find allowing 
 filtering beneficial for their workloads so I think removing it makes 
 sense to me (and yes we try to discourage them 🙂)
 
 I am also intrigued by this proposal when I think about multi tenancy and 
 resource governance: We have heard from several operator who run multiple 
 internal teams on the same Cassandra cluster jut to optimize costs. Having 
 a way to attribute those costs more fairly by adding up the costs the 
 optimizer calculates might be hugely beneficial.  There could also be a 
 way to have a "cost budget" on a keyspace to minimize the noisy neighbor 
 problem and do more intelligent request throttling. 
 
 In summary I support the proposal with the caveats raised above.
 
 Thanks,
 German
 
 
 *From:* C. Scott Andreas 
 *Sent:* Wednesday, December 20, 2023 8:15 AM
 *To:* dev@cassandra.apache.org 
 *Cc:* dev@cassandra.apache.org 
 *Subject:* [EXTERNAL] Re: [DISCUSS] CEP-39: Cost Based Optimizer
  
 
 You don't often get email from sc...@paradoxica.net. Learn why this is 
 important 
 

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[J. D. Jordan]

The CEP-29 “rejected alternatives” section mentions one such use case.  Being able to put NOT arbitrarily in a query.  Adding an OR operator is another thing we are likely to want to do in the near future that would benefit from this work, those benefit from the syntax tree and reordering parts of the proposal.But I think we already have enough complexity available to us to justify a query optimizer in the fact of multi index queries today. Especially when you have the new ANN OF operator in use combined with index queries.  Depending on what order you query the indexes in, it can dramatically change the performance of the query.  We are seeing and working through such issues in Astra today.-JeremiahOn Dec 21, 2023, at 12:00 PM, Josh McKenzie  wrote:we are already late. We have several features running in production that we chose to not open source yet because implementing phase 1 of the CEP would have heavily simplify their designs. The cost of developing them was much higher than what it would have been if the CEP had already been implemented. We are also currently working on some SAI features that need cost based optimization.Are there DISCUSS threads or CEP's for any of that work? For us to have a useful discussion about whether we're at a point in the project where a query optimizer is appropriate for the project this information would be vital.On Thu, Dec 21, 2023, at 12:33 PM, Benjamin Lerer wrote:Hey German,To clarify things, we intend to push cardinalities across nodes, not costs. It will be up to the Cost Model to estimate cost based on those cardinalities. We will implement some functionalities to collect costs on query execution to be able to provide them as the output of EXPLAIN ANALYZE.We will provide more details on how we will collect and distribute cardinalities. We will probably not go into details on how we will estimate costs before the patch for it is ready. The main reason being that there are a lot of different parts that you need to account for and that it will require significant testing and experimentation.Regarding multi-tenancy, even if you use query cost, do not forget that you will have to account also for background tasks such as compaction, repair, backup, ... which is not included in this CEP.      Le jeu. 21 déc. 2023 à 00:18, German Eichberger via dev  a écrit :All,very much agree with Scott's reasoning. It seems expedient given the advent of ACCORD transactions to be more like the other distributed SQL databases and just support SQL. But just because it's expedient it isn't right and we should work out the relational features in more detail before we embark
 on tying us to some query planning design.The main problem in this space is pushing cost / across nodes based on data density.
 I understand that TCM will level out data density but the cost based optimizer proposal does a lot of hand waving when it comes to collecting/estimating costs for each node. I like to see more details on this since otherwise it will be fairly limiting.I am less tied to ALLOW FILTERING - many of my customers find allowing filtering beneficial
 for their workloads so I think removing it makes sense to me (and yes we try to discourage them 🙂)I am also intrigued by this proposal when I think about multi tenancy and resource governance:
 We have heard from several operator who run multiple internal teams on the same Cassandra cluster jut to optimize costs. Having a way to attribute those costs more fairly by adding up the costs the optimizer calculates might be hugely beneficial.  There could
 also be a way to have a "cost budget" on a keyspace to minimize the noisy neighbor problem and do more intelligent request throttling. In summary I support the proposal with the caveats raised above.Thanks,GermanFrom: C. Scott Andreas  Sent: Wednesday, December 20, 2023 8:15 AM To: dev@cassandra.apache.org  Cc: dev@cassandra.apache.org  Subject: [EXTERNAL] Re: [DISCUSS] CEP-39: Cost Based Optimizer You don't often get email from sc...@paradoxica.net.  Learn why this is importantThanks for this proposal and apologies for my delayed engagement during the Cassandra Summit last week. Benjamin, I appreciate your work on this and your engagement on this thread – I know it’s a lot of discussion to field.On ALLOW FILTERING:I share Chris Lohfink’s experience in operating clusters that have made heavy use of ALLOW FILTERING. It is a valuable guardrail for the database to require users specifically annotate queries that may cost 1000x+ that of a simple lookup for a primary
 key. For my own purposes, I’d actually like to go a step further and disable queries that require ALLOW FILTERING by default unless explicitly reviewed - but haven’t taken the step of adding such a guardrail yet.CBOs, CQL, and SQL:The CBO proposal cuts to the heart of one of the fundamental differences between SQL and CQL that I haven’t seen exercis

Re: [DISCUSS] CEP-39: Cost Based Optimizer

[Benedict]

Close enough, though that’s not quite how I would characterise it. But none of your problems are inherent?- Clients can re-prepare whenever they want- Clusters can suggest to clients to re-prepare, should we desire this feature. Or we could permit the cluster to invalidate stale preparations since the query+plan are both part of the id, forcing a client to re-prepare.Your fourth characteristic is not unique to this proposal.My main concern however is avoiding this coordinated cardinality statistics problem. I’m not keen on introducing a new complex whole cluster maintenance process. This is costly to maintain in both developer and system resources. At the very least it feels premature for the value proposition today, as others have said, whereas the minimal version of what I propose requires little work to get moving - just the ability to serialise execution plans, a small client protocol change and a minor tweak to prepared statement hashing.If we ever care about solving consistent execution we will have to do this anyway, so we both simplify initial delivery while better preparing for the future. We also for free introduce a mechanism for users that want deterministic execution to expressly specify how their query will execute.On 20 Dec 2023, at 17:26, Benjamin Lerer  wrote:
Pick a random replica and ask it to prepare it, and use that. This
 is probably fine, unless there is significant skew (in which case, our 
plan is likely to bad whatever, somewhere)To be sure that I understand you correctly. What you are suggesting is to use local statistics to compute the execution plan and then rely on the driver performing the query to cache it?If it is the case, then it would mean that:a bad execution plan will only be corrected on client restartif the data distribution change the driver will keep on using its plandifferent drivers might use different plans2 similar queries that differ on simple ways might have totally different plansDo I understand your proposal and its trades off correctly?

Le mer. 20 déc. 2023 à 17:16, Benedict  a écrit :I see three options:Pick a random replica and ask it to prepare it, and use that. This is probably fine, unless there is significant skew (in which case, our plan is likely to bad whatever, somewhere)If there already exists a plan for the query, return thatPick a sample of replicas and eitherAsk them for their plan, and pick the most common planAsk them for their cardinality estimates, and use some combination thereof The nice thing is that this can be evolved independently of the rest of the proposal, as the simple option is fine. But also we don’t have to introduce any new whole cluster operations - any work we might perform is done only on statement preparation, and only for data relevant to the statement that is being prepared, and only on a minority of shardsand one replica per shard (in a single DC), rather than on some arbitrary schedule for every replica.On 20 Dec 2023, at 15:52, Benjamin Lerer  wrote:
If we are to address that within the CEP itself then we should discuss 
it here, as I would like to fully understand the approach as well as how
 it relates to consistency of execution and the idea of triggering 
re-optimisation.

Sure, that was my plan.
I’m not sold on the proposed set of characteristics, and 
think my coupling an execution plan to a given prepared statement for 
clients to supply is perhaps simpler to implement and maintain, and has 
corollary benefits - such as providing a mechanism for users to specify 
their own execution plan.  Note,
 my proposal cuts across all of these elements of the CEP. There is no 
obvious need for a cross-cluster re-optimisation event or cross cluster 
statistic management.  I think that I am missing one part of your proposal. How do you plan to build the initial execution plan for a prepared statement?Le mer. 20 déc. 2023 à 14:05, Benedict  a écrit :If we are to address that within the CEP itself then we should discuss it here, as I would like to fully understand the approach as well as how it relates to consistency of execution and the idea of triggering re-optimisation. These ideas are all interrelated.I’m not sold on the proposed set of characteristics, and think my coupling an execution plan to a given prepared statement for clients to supply is perhaps simpler to implement and maintain, and has corollary benefits - such as providing a mechanism for users to specify their own execution plan.Note, my proposal cuts across all of these elements of the CEP. There is no obvious need for a cross-cluster re-optimisation event or cross cluster statistic management.We still also need to discuss more concretely how the base statistics themselves will be derived, as there is little detail here today in the proposal.On 20 Dec 2023, at 12:58, Benjamin Lerer  wrote:After the second phase of the CEP, we will have two optimizer implementations. One wil

Re: [DISCUSSION] Framework for Internal Collection Exposure and Monitoring API Alignment

[Maxim Muzafarov]

Hello everyone and happy holidays,

The changes below are ready for review!
Benchmarks are also inside.

Expose all table metrics in virtual tables
https://issues.apache.org/jira/browse/CASSANDRA-14572
https://github.com/apache/cassandra/pull/2958/files

On Tue, 12 Dec 2023 at 22:05, Maxim Muzafarov  wrote:
>
> Hello everyone,
>
>
> I still think Cassandra will benefit from having this idea implemented
> and used through the source code, so I've done another round of
> rethinking this concept and it seems I've found a solution. As a
> result, we can significantly reduce the cost of implementing and
> maintaining both new and existing virtual tables and make our users
> happier by seeing everything they need through virtual tables.
>
> So, I think we should limit the scope of the original proposal to the 
> following:
> ## A framework for exposing any internal data collection to virtual
> tables ONLY. ##
>
> As a proof of concept, I took the CASSANDRA-14572 "Expose all table
> metrics in virtual table" JIRA ticket, which provides a good
> opportunity to demonstrate how to export all metrics to VTs at once
> without having boilerplate implementations. Currently, we already have
> CQLMetricsTable, BatchMetricsTable, etc. that expose metrics to VTs in
> a pretty similar way, and most of the metrics groups are located under
> the org.apache.cassandra.metrics package still lacks their
> representation as VTs either. I've used the MetricRegistry collection
> as a view of registered metrics to export them to VT using the
> prototype accordingly.
>
> The prototype is complete. You can run a node locally and check the
> available virtual tables with cqlsh, or you can check the changes
> using the following link to the PR:
> https://github.com/apache/cassandra/pull/2958/files
>
>
> Below are some key points about the design itself:
>
> 1. All new virtual tables with metrics have "metric" as a prefix so
> that they are fairly easy to find using TAB on the cqlsh command line.
> Metrics are split into virtual tables as they are listed in the
> org.apache.cassandra.metrics e.g. metrics_cql, metrics_tcm etc. In
> addition, they are also grouped by metric type e.g.
> metric_type_histogram, metric_type_counter etc. There is a table
> called "metric_all_metric_groups" with all available metric groups.
>
> 2. To create a new virtual table representation of an internal
> collection a developer needs to do two things: create a virtual table
> row representation, and register it using
> CollectionVirtualTableAdapter, which acts as an adapter between
> internal data and a virtual table. Here's how I did it for the thread
> pools VT, this is a fully backward compatible change:
> https://github.com/apache/cassandra/pull/2958/files#diff-5fda13a633723cdf232bba465e6fb7ab74cdc02f7ba55dae4d1cf494ffb71abaR61
>
> 3. The "metrics_keyspace" virtual table ended up being quite large
> since it contains all the metrics for all available keyspaces on a
> local node, so the default implementation provided by
> AbstractVirtualTable is not suitable for the proposal. The
> AbstractVirtualTable materializes a full data collection on the heap
> using SimpleDataSet, regardless of the portion of data that is being
> queried. In this case, we have to use an iterative approach, as the
> CollectionVirtualTableAdapter does (the problem was discussed in
> CASSANDRA-14629 and is now a part of the solution). This also helps to
> keep the memory footprint low.
>
> 4. Another valuable change is the CassandraMetricsRegistry itself. The
> problem here is that the metrics and their aliases are currently
> exported to JMX, but the implemented virtual tables export the metrics
> in their way and most of the cases don't respect the metric aliases
> which are registered in the MetricsRegistry. This should be fixed as a
> part of the CASSANDRA-14572 to avoid ambiguity for all known metrics
> once and for all.
>
> Here are the links to the issue and the PR:
> https://issues.apache.org/jira/browse/CASSANDRA-14572
> https://github.com/apache/cassandra/pull/2958/files
>
>
> I'm excited about how these changes look right now, so please share
> your feedback and thoughts.
> The PR lacks good test coverage, I'll fix it as soon as we have a
> clear vision of the design (or much sooner) :-)
>
> On Mon, 30 Jan 2023 at 17:43, David Capwell  wrote:
> >
> > I *think* this is arguably true for a vtable / CQL-based solution as well 
> > from the "you don't know how people are using your API" perspective.
> >
> >
> > Very fair point and think that justifies a different thread to talk about 
> > backwards compatibility for our tables (virtual and not); maybe we can lump 
> > together the JMX backwards compatibility conversation as well in that new 
> > thread.
> >
> > On Jan 28, 2023, at 4:09 PM, Josh McKenzie  wrote:
> >
> > First off - thanks so much for putting in this effort Maxim! This is 
> > excellent work.
> >
> > Some thoughts on the CEP and responses in thread:
> >
> > Consid

Re: Harry in-tree (Forked from "Long tests, Burn tests, Simulator tests, Fuzz tests - can we clarify the diffs?")

[Alex Petrov]

Some follow-up tickets to establish the project direction:

https://issues.apache.org/jira/browse/CASSANDRA-19229

Two other things that we will work on in Tree are:
https://issues.apache.org/jira/browse/CASSANDRA-18275 (model and in-JVM test 
for partition-restricted 2i queries) 
https://issues.apache.org/jira/browse/CASSANDRA-18667 (multi-threaded SAI read 
and write fuzz test)

If you would like to get your recently added feature tested with Harry model, 
please let me know!

On Fri, Dec 22, 2023, at 12:41 AM, Joseph Lynch wrote:
> +1
> 
> Sounds like a great change that will help us unify around a common testing 
> paradigm, and even pave the path to in-tree load testing plus integrated 
> correctness checking which would be extremely valuable!
> 
> -Joey
> 
> On Thu, Dec 21, 2023 at 1:35 PM Caleb Rackliffe  
> wrote:
>> +1
>> 
>> Agree w/ all the justifications mentioned above.
>> 
>> As a reviewer on CASSANDRA-19210 
>> , my goals were to 
>> a.) look at the directory, naming, and package structure of the ported code, 
>> b.) make sure IDE integration was working, and c.) make sure any 
>> modifications to existing code (rather than direct code movements from 
>> cassandra-harry) were straightforward.
>> 
>> On Thu, Dec 21, 2023 at 3:23 PM Alex Petrov  wrote:
>>> __
>>> Hey folks,
>>> 
>>> I am mostly done with a patch that brings Harry in-tree [1]. I will trigger 
>>> one more CI run overnight, and my intention was to merge it some time soon, 
>>> but I wanted to give a fair warning here, since this is a relatively large 
>>> patch. 
>>> 
>>> Good news for everyone that it: 
>>>   a) touches no production code whatsoever. Only test (in-jvm dtest namely) 
>>> code that was using Harry already.
>>>   b) the only tests that are changed are ones that used a duplicate version 
>>> of placement simulator we had both for testing TCM, and in Harry
>>>   c) in addition, I have converted 3 existing TCM tests to a new API to 
>>> have some base for examples/usage.
>>> 
>>> Since we were effectively relying on this code for a while now, and the 
>>> intention now is to converge to: 
>>>   a) fewer different generators, and have a shareable version of generators 
>>> for everyone to use accross the base
>>>   b) a testing tool that can be useful for both trivial cases, and complex 
>>> scenarios 
>>> myself and many other Cassandra contributors have expressed an opinion that 
>>> bringing Harry in-tree will be highly benefitial.
>>> 
>>> I strongly believe that bringing Harry in-tree will help to lower the 
>>> barrier for fuzz test and simplify co-development of Cassandra and Harry. 
>>> Previously, it has been rather difficult to debug edge cases because I had 
>>> to either re-compile an in-jvm dtest jar and bring it to Harry, or 
>>> re-compile a Harry jar and bring it to Cassandra, which is both tedious and 
>>> time consuming. Moreover, I believe we have missed at very least one RT 
>>> regression [2] because Harry was not in-tree, as its tests would've caught 
>>> the issue even with the model that existed.
>>> 
>>> For other recently found issues, I think having Harry in-tree would have 
>>> substantially lowered a turnaround time, and allowed me to share repros 
>>> with developers of corresponding features much quicker.
>>> 
>>> I do expect a slight learning curve for Harry, but my intention is to build 
>>> a web of simple tests (worked on some of them yesterday after conversation 
>>> with David already), which can follow the in-jvm-dtest pattern of 
>>> find-similar-test / copy / modify. There's already copious documentation, 
>>> so I do not believe not having docs for Harry was ever an issue, since 
>>> there have been plenty. 
>>> 
>>> You all are aware of my dedication to testing and quality of Apache 
>>> Cassandra, and I hope you also see the benefits of having a model checker 
>>> in-tree.
>>> 
>>> Thank you and happy upcoming holidays,
>>> --Alex
>>> 
>>> [1] https://issues.apache.org/jira/browse/CASSANDRA-19210
>>> [2] https://issues.apache.org/jira/browse/CASSANDRA-18932
>>>