Re: Welcome Alexandre Dutra, Andrew Tolbert, Bret McGuire, Olivier Michallat as Cassandra Committers

[Jeremy Hanna]

Congratulations to all and it’s exciting to see the java driver get its footing 
with some great contributors/committers.

> On Apr 17, 2024, at 12:14 PM, Abe Ratnofsky  wrote:
> 
> Congrats everyone!
> 
>> On Apr 17, 2024, at 1:10 PM, Benjamin Lerer  wrote:
>> 
>> The Apache Cassandra PMC is pleased to announce that Alexandre Dutra, Andrew 
>> Tolbert, Bret McGuire and Olivier Michallat have accepted the invitation to 
>> become committers on the java driver sub-project. 
>> 
>> Thanks for your contributions to the Java driver during all those years!
>> Congratulations and welcome!
>> 
>> The Apache Cassandra PMC members
> 

Re: Welcome Brad Schoening as Cassandra Committer

[Jeremy Hanna]

Congratulations Brad!

> On Feb 21, 2024, at 3:59 PM, Leo Toff  wrote:
> 
> Congratulations Brad! Thank you for helping me onboard 
> 
> On Wed, Feb 21, 2024 at 1:56 PM Jeremiah Jordan  > wrote:
>> Congrats!
>> 
>> On Feb 21, 2024 at 2:46:14 PM, Josh McKenzie > > wrote:
>>> The Apache Cassandra PMC is pleased to announce that Brad Schoening has 
>>> accepted
>>> the invitation to become a committer.
>>> 
>>> Your work on the integrated python driver, launch script environment, and 
>>> tests
>>> has been a big help to many. Congratulations and welcome!
>>> 
>>> The Apache Cassandra PMC members

Re: [VOTE] Release Apache Cassandra 5.0-beta1

[Jeremy Hanna]

I want to just follow up with functional versus production-worthy.  If I'm a 
user interested in C* 5 and want to try it out as betas come out, I'm looking 
more for something functional and not perfect.  So in the context of this 
thread, if I want to try out SAI for example, I don't care as much about 
consistency edge cases around coordinators or replicas or read repair.  I care 
a lot about that for a RC or GA release but doing POCs with betas that have 
known edge case issues like that is fine IMO.

I know this is likely a moot point for this release since the fixes are almost 
in, but I think just publishing beta 1 and then a follow up beta 2 with those 
fixes would be fine in that context, if I understood the bugs correctly.

> On Nov 29, 2023, at 12:15 PM, Aaron Ploetz  wrote:
> 
> Even though my opinion doesn't really count here, I do feel compelled to 
> mention that:
> 
>  - No one expects a "beta" release to be perfect, but it does signal that it 
> is "close" to being ready.
>  - An "alpha" release is in fact a LOT scarier than a "beta" release.
> 
> From a user perspective, if I was coaching dev teams on selecting a build 
> based on newly available features, I would help them build up a dev/stage 
> cluster based on a beta (and make the "beta" part very clear to them). 
> However an alpha version just doesn't convey the same level of confidence. 
> When I test out an "alpha" of anything, I fully expect some things to just be 
> broken.
> 
> As for cutting a beta for the Summit; it makes sense that we'd want to get 
> some things fixed up before that. But it would also be great to be at the 
> point where we have a beta ready for folks to take a look at. We absolutely 
> could tell everyone to download the alpha and give it a spin. But more people 
> will be likely to do that for a beta than for an alpha.
> 
> Take that however you will.
> 
> Thanks,
> 
> Aaron
> 
> 
> On Wed, Nov 29, 2023 at 9:54 AM Aleksey Yeshchenko  > wrote:
>> -1 on cutting a beta1 in this state. An alpha2 would be acceptable now, but 
>> I’m not sure there is significant value to be had from it. Merge the fixes 
>> for outstanding issues listed above, then cut beta1.
>> 
>> With TCM and Accord pushed into 5.1, SAI is the headliner user-visible 
>> feature. It is what we want users to test. If we are to drive more people to 
>> test SAI, we should resolve the issues that we ourselves know of first. 
>> Respect our users’ time and effort - don’t make them bump into known bugs.
>> 
>> P.S. I don’t believe that ‘alpha' vs. ‘beta' really makes a significant 
>> difference to people’s willingness to try out the build. For most folks both 
>> feel too raw to play with, and most of the rest would be equally adventurous 
>> enough for an alpha *or* a beta. This is just my gut feeling vs. your gut 
>> feeling, in absence of hard data.
>> 
>>> On 28 Nov 2023, at 21:17, Mick Semb Wever >> > wrote:
>>> 
>>> 
>>> 
 So then cutting an alpha2 is possible.
>>> 
>>> 
>>> Possible, but still leaves alpha1 as our mitigation plan and alpha2 as our 
>>> best plan.  Doesn't seem worth it IMHO.
>> 

Re: [DISCUSS] Harry in-tree

[Jeremy Hanna]

I'm excited for Harry to come in-tree to improve the project stability and 
quality.  I know you're doing a talk at the Cassandra Summit about Harry to go 
over it.  If there's anything that can be done as part of this process to 
improve onboarding for Harry too, that would be great.  I'm just thinking about 
examples and things like that so people new to Harry can more easily write and 
run tests, test new features, and have a standard process for reporting 
findings.

Thanks Alex and all involved!

Jeremy

> On Nov 24, 2023, at 9:43 AM, Alex Petrov  wrote:
> 
> Hi everyone,
> 
> With TCM landed, there will be way more Harry tests in-tree: we are using it 
> for many coordination tests, and there's now a simulator test that uses 
> Harry. During development, Harry has allowed us to uncover and resolve 
> numerous elusive edge cases.
> 
> I had conversations with several folks, and wanted to propose to move 
> harry-core to Cassandra test tree. This will substantially 
> simplify/streamline co-development of Cassandra and Harry. With a new 
> HistoryBuilder API that has helped to find and trigger [1] [2] and [3], it 
> will also be much more approachable.
> 
> Besides making it easier for everyone to develop new fuzz tests, it will also 
> substantially lower the barrier to entry. Currently, debugging an issue found 
> by Harry involves a cumbersome process of rebuilding and transferring jars 
> between Cassandra and Harry, depending on which side you modify. This not 
> only hampers efficiency but also deters broader adoption. By merging 
> harry-core into the Cassandra test tree, we eliminate this barrier.
> 
> Thank you,
> --Alex
> 
> [1] https://issues.apache.org/jira/browse/CASSANDRA-19011
> [2] https://issues.apache.org/jira/browse/CASSANDRA-18993
> [3] https://issues.apache.org/jira/browse/CASSANDRA-18932

Re: Push TCM (CEP-21) and Accord (CEP-15) to 5.1 (and cut an immediate 5.1-alpha1)

[Jeremy Hanna]

I think the goal is to say "how could we get some working version of TCM/Accord 
into people's hands to try out at/by Summit?"  That's all.  People are eager to 
see it and try it out.

> On Oct 31, 2023, at 12:16 PM, Benedict  wrote:
> 
> No, if I understand it correctly we’re in weird hypothetical land where 
> people are inventing new release types (“preview”) to avoid merging TCM[1] in 
> the event we want to cut a 5.1 release from the PR prior to the summit if 
> there’s some handful of failing tests in the PR. 
> 
> This may or may not be a waste of everyone’s time.
> 
> Jeremiah, I’m not questioning: it was procedurally invalid. How we handle 
> that is, as always, a matter for community decision making.
> 
> [1] how this helps isn’t entirely clear
> 
>> On 31 Oct 2023, at 17:08, Paulo Motta  wrote:
>> 
>> 
>> Even if it was not formally prescribed as far as I understand, we have been 
>> following the "only merge on Green CI" custom as much as possible for the 
>> past several years. Is the proposal to relax this rule for 5.0?
>> 
>> On Tue, Oct 31, 2023 at 1:02 PM Jeremiah Jordan > > wrote:
>>> You are free to argue validity.  I am just stating what I see on the 
>>> mailing list and in the wiki.  We had a vote which was called passing and 
>>> was not contested at that time.  The vote was on a process which includes 
>>> as #3 in the list:
>>> 
>>> Before a merge, a committer needs either a non-regressing (i.e. no new 
>>> failures) run of circleci with the required test suites (TBD; see below) or 
>>> of ci-cassandra.
>>> Non-regressing is defined here as "Doesn't introduce any new test failures; 
>>> any new failures in CI are clearly not attributable to this diff"
>>> (NEW) After merging tickets, ci-cassandra runs against the SHA and the 
>>> author gets an advisory update on the related JIRA for any new errors on 
>>> CI. The author of the ticket will take point on triaging this new failure 
>>> and either fixing (if clearly reproducible or related to their work) or 
>>> opening a JIRA for the intermittent failure and linking it in butler 
>>> (https://butler.cassandra.apache.org/#/)
>>> 
>>> Which clearly says that before merge we ensure there are no known new 
>>> regressions to CI.
>>> 
>>> The allowance for releases without CI being green, and merges without the 
>>> CI being completely green are from the fact that our trunk CI has rarely 
>>> been completely green, so we allow merging things which do not introduce 
>>> NEW regressions, and we allow releases with known regressions that are 
>>> deemed acceptable.
>>> 
>>> We can indeed always vote to override it, and if it comes to that we can 
>>> consider that as an option.
>>> 
>>> -Jeremiah
>>> 
>>> On Oct 31, 2023 at 11:41:29 AM, Benedict >> > wrote:
 That vote thread also did not reach the threshold; it was incorrectly 
 counted, as committer votes are not binding for procedural changes. I 
 counted at most 8 PMC +1 votes. 
 
 The focus of that thread was also clearly GA releases and merges on such 
 branches, since there was a focus on releases being failure-free. But this 
 predates the more general release lifecycle vote that allows for alphas to 
 have failing tests - which logically would be impossible if nothing were 
 merged with failing or flaky tests.
 
 Either way, the vote and discussion specifically allow for this to be 
 overridden.
 
 路‍♀️
 
> On 31 Oct 2023, at 16:29, Jeremiah Jordan  > wrote:
> 
> 
> I never said there was a need for green CI for alpha.  We do have a 
> requirement for not merging things to trunk that have known regressions 
> in CI.
> Vote here: 
> https://lists.apache.org/thread/j34mrgcy9wrtn04nwwymgm6893h0xwo9
> 
> 
> 
> On Oct 31, 2023 at 3:23:48 AM, Benedict  > wrote:
>> There is no requirement for green CI on alpha. We voted last year to 
>> require running all tests before commit and to require green CI for beta 
>> releases. This vote was invalid because it didn’t reach the vote floor 
>> for a procedural change but anyway is not inconsistent with knowingly 
>> and selectively merging work without green CI.
>> 
>> If we reach the summit we should take a look at the state of the PRs and 
>> make a decision about if they are alpha quality; if so, and we want a 
>> release, we should simply merge it and release. Making up a new release 
>> type when the work meets alpha standard to avoid an arbitrary and not 
>> mandated commit bar seems the definition of silly.
>> 
>>> On 31 Oct 2023, at 04:34, J. D. Jordan >> > wrote:
>>> 
>>> 
>>> That is my understanding as well. If the TCM and Accord based on TCM 
>>> branches are ready to commit by ~12/1 we can 

Re: Status Update on CEP-7 Storage Attached Indexes (SAI)

[Jeremy Hanna]

Thanks Caleb and Mike and Zhao and Andres and Piotr and everyone else involved with the SAI implementation!On Jul 25, 2023, at 3:01 PM, Caleb Rackliffe  wrote:Just a quick update...With CASSANDRA-18670 complete, and all remaining items in the category of performance optimizations and further testing, the process of merging to trunk will likely start today, beginning with a final rebase on the current trunk and J11 and J17 test runs.On Tue, Jul 18, 2023 at 3:47 PM Caleb Rackliffe  wrote:Hello there!After much toil, the first phase of CEP-7 is nearing completion (see CASSANDRA-16052). There are presently two issues to resolve before we'd like to merge the cep-7-sai feature branch and all its goodness to trunk:CASSANDRA-18670 - Importer should build SSTable indexes successfully before making new SSTables readable (in review)CASSANDRA-18673 - Reduce size of per-SSTable index components (in progress)(We've been getting clean CircleCI runs for a while now, and have been using the multiplexer to sniff out as much flakiness as possible up front.)Once merged to trunk, the next steps are:1.) Finish a Harry model that we can use to further fuzz test SAI before 5.0 releases (see CASSANDRA-18275). We've done a fair amount of fuzz/randomized testing at the component level, but I'd still consider Harry (at least around single-partition query use-cases) a critical item for us to have confidence before release.2.) Start pursuing Phase 2 items as time and our needs allow. (see CASSANDRA-18473)A reminder, SAI is a secondary index, and therefore is by definition an opt-in feature, and has no explicit "feature flag". However, its availability to users is still subject to the secondary_indexes_enabled guardrail, which currently defaults to allowing creation.Any thoughts, questions, or comments on the pre-merge plan here?

Re: [VOTE] CEP 33 - CIDR filtering authorizer

[Jeremy Hanna]

+1 (nb) will be great to get this into the project.

> On Jun 28, 2023, at 12:15 PM, Patrick McFadin  wrote:
> 
> +1
> 
> On Wed, Jun 28, 2023 at 3:42 AM Brandon Williams  > wrote:
>> +1
>> 
>> Kind Regards,
>> Brandon
>> 
>> 
>> On Tue, Jun 27, 2023 at 12:17 PM Shailaja Koppu > > wrote:
>> >
>> > Hi Team,
>> >
>> > (Starting a new thread for VOTE instead of reusing the DISCUSS thread, to 
>> > follow usual procedure).
>> >
>> > Please vote on CEP 33 - CIDR filtering authorizer 
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-33%3A+CIDR+filtering+authorizer.
>> >
>> > Thanks,
>> > Shailaja

Re: [DISCUSS] Being specific about JDK versions and python lib versions in CI

[Jeremy Hanna]

I like having the latest patch release always brought in for testing and CI for the JDK versions explicitly supported. So for this release, 11/17.On Jun 22, 2023, at 5:42 PM, Jeremiah Jordan  wrote:
Yes.  -1 on forcing the patch release version, and possibly minor version, for anything that is not absolutely necessary to do so.  Especially for things like Java or Python version, I would hope we just install the latest Java 8, Java 11, or Java 17 JDK for the platform the image is built from and run under them.  Otherwise we don’t find out until it’s too late when some JDK update breaks things.  I know I always tell users to run the latest JDK patch release, so we should do the same.If we want to pin the major version of something, then I have no problem there.-Jeremiah


On Jun 22, 2023 at 5:00:36 PM, Ekaterina Dimitrova  wrote:

Wouldn’t we recommend people to use the test images the project CI use? Thus using in testing the versions we use? I would assume the repeatable CI will still expect test images the way we have now? (I hope I did not misunderstand the question)I also share similar worries with BrandonOn Thu, 22 Jun 2023 at 17:48, Brandon Williams  wrote:On Thu, Jun 22, 2023 at 4:23 PM Josh McKenzie  wrote:
>
> 2. Anyone concerned about us being specific about versions in requirements.txt in the dtest repo?

My concern with this is atrophy; we'll set the version once and when
finally forced to update, find that a lot of other things must also be
updated in order to do so.  I think our current approach of only
setting them on things we require at a certain version like thrift has
been successful thus far, and I don't think having different versions
would be very common, but also not really affect repeatability if
encountered.  You can see what versions are used from the logs though
and could adjust them to be the same if necessary.

Re: [VOTE] CEP-8 Datastax Drivers Donation

[Jeremy Hanna]

After 72 hours, this is the summary of the voting:

binding +1 votes: 21
non-binding +1 votes: 9
-1 votes: 0
vetoes: 0

It looks like CEP-8 has passed at long last.

Thanks everyone!

Jeremy

> On Jun 16, 2023, at 6:04 AM, Aleksey Yeshchenko  wrote:
> 
> +1
> 
>> On 15 Jun 2023, at 15:19, Chris Lohfink  wrote:
>> 
>> +1
>> 
>> On Wed, Jun 14, 2023 at 9:05 PM Jon Haddad > <mailto:rustyrazorbl...@apache.org>> wrote:
>>> +1
>>> 
>>> On 2023/06/13 14:14:35 Jeremy Hanna wrote:
>>> > Calling for a vote on CEP-8 [1].
>>> > 
>>> > To clarify the intent, as Benjamin said in the discussion thread [2], the 
>>> > goal of this vote is simply to ensure that the community is in favor of 
>>> > the donation. Nothing more.
>>> > The plan is to introduce the drivers, one by one. Each driver donation 
>>> > will need to be accepted first by the PMC members, as it is the case for 
>>> > any donation. Therefore the PMC should have full control on the pace at 
>>> > which new drivers are accepted.
>>> > 
>>> > If this vote passes, we can start this process for the Java driver under 
>>> > the direction of the PMC.
>>> > 
>>> > Jeremy
>>> > 
>>> > 1. 
>>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
>>> > 2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp
> 

[VOTE] CEP-8 Datastax Drivers Donation

[Jeremy Hanna]

Calling for a vote on CEP-8 [1].

To clarify the intent, as Benjamin said in the discussion thread [2], the goal 
of this vote is simply to ensure that the community is in favor of the 
donation. Nothing more.
The plan is to introduce the drivers, one by one. Each driver donation will 
need to be accepted first by the PMC members, as it is the case for any 
donation. Therefore the PMC should have full control on the pace at which new 
drivers are accepted.

If this vote passes, we can start this process for the Java driver under the 
direction of the PMC.

Jeremy

1. 
https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation
2. https://lists.apache.org/thread/opt630do09phh7hlt28odztxdv6g58dp

Re: [DISCUSS] CEP-8 Drivers Donation - take 2

[Jeremy Hanna]

I'd like to close out this thread.  As Benjamin notes, we'll have a single 
subproject for all of the drivers and with 3 PMC members overseeing the 
subproject as outlined in the linked subproject governance procedures.  However 
we'll introduce the drivers to that subproject one by one out of necessity.

I'll open up a vote thread shortly so that we can move forward on the CEP and 
subproject approach.

> On May 30, 2023, at 7:32 AM, Benjamin Lerer  wrote:
> 
> The idea was to have a single driver sub-project. Even if the code bases are 
> different we believe that it is important to keep the drivers together to 
> retain cohesive API semantics and make sure they have similar functionality 
> and feature support.
> In this scenario we would need only 3 PMC members for the governance. I am 
> willing to be one of them.
> 
> For the committers, my understanding, based on subproject governance 
> procedures, 
>  was that 
> they should be proposed directly to the PMC members.
> 
>> Is the vote for the CEP to be for all drivers, but we will introduce each 
>> driver one by one?  What determines when we are comfortable with one driver 
>> subproject and can move on to accepting the next ? 
> 
> The goal of the CEP is simply to ensure that the community is in favor of the 
> donation. Nothing more. 
> The plan is to introduce the drivers, one by one. Each driver donation will 
> need to be accepted first by the PMC members, as it is the case for any 
> donation. Therefore the PMC should have full control on the pace at which new 
> drivers are accepted.
>   
> 
> Le mar. 30 mai 2023 à 12:22, Josh McKenzie  > a écrit :
>>> Is the vote for the CEP to be for all drivers, but we will introduce each 
>>> driver one by one?  What determines when we are comfortable with one driver 
>>> subproject and can move on to accepting the next ? 
>> Curious to hear on this as well. There's 2 implications from the CEP as 
>> written:
>> 
>> 1. The Java and Python drivers hold special importance due to their language 
>> proximity and/or project's dependence upon them 
>> (https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation#CEP8:DatastaxDriversDonation-Scope)
>> 2. Datastax is explicitly offering all 7 drivers for donation 
>> (https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation#CEP8:DatastaxDriversDonation-Goals)
>> 
>> This is the most complex contribution via CEP thus far from a governance 
>> perspective; I suggest we chart a bespoke path to navigate this. Having a 
>> top level indication of "the CEP is approved" logically separate from a 
>> per-language indication of "the project is ready to absorb this language 
>> driver now" makes sense to me. This could look like:
>> 
>> * Vote on the CEP itself
>> * Per language (processing one at a time):
>> * identify 3 PMC members willing to take on the governance role for the 
>> language driver
>> * Identify 2 contributors who are active on a given driver and stepping 
>> forward for a committer role on the driver
>> * Vote on inclusion of that language driver in the project + commit bits
>> * Integrate that driver into the project ecosystem (build, ci, docs, etc)
>> 
>> Not sure how else we could handle committers / contributors / PMC members 
>> other than on a per-driver basis.
>> 
>> On Tue, May 30, 2023, at 5:36 AM, Mick Semb Wever wrote:
>>> 
>>> Thank you so much Jeremy and Greg (+others) for all the hard work on this.
>>>  
>>> 
>>> At this point, we'd like to propose CEP-8 for consideration, starting the 
>>> process to accept the DataStax Java driver as an official ASF project.
>>> 
>>> 
>>> Is the vote for the CEP to be for all drivers, but we will introduce each 
>>> driver one by one?  What determines when we are comfortable with one driver 
>>> subproject and can move on to accepting the next ? 
>>> 
>>> Are there key committers and contributors on each driver that want to be 
>>> involved?  Should they be listed before the vote?
>>> We also need three PMC for the new subproject.  Are we to assign these 
>>> before the vote?  
>>> 
>>> 
>> 

[DISCUSS] CEP-8 Drivers Donation - take 2

[Jeremy Hanna]

To add to a somewhat crowded [DISCUSS] party, I'd like to restart the 
discussion around CEP-8.

This is the original thread from July 2020: 
https://lists.apache.org/thread/01pljcncyjyo467l5orh8nf9okrh7oxm

At the time, several good points were discussed and the CEP has been updated 
with many of them.  One point in particular was that we should start with the 
DataStax Java driver as it is the reference implementation of the CQL protocol, 
a dependency of the project, and the most used of the 7 drivers discussed.  
Other points were about package naming evolution and DataStax specific 
functionality.  I believe everyone agreed that we should take the first step of 
contributing the drivers as-is to minimize user disruption.  That way we get 
through the legal and procedural process for the first driver.  Then we can 
proceed with discussing how it will be managed and by whom.

As the next step in donating the Java driver to the project and as we talked 
about at ApacheCon last year, we needed to verify that we had all of the CLAs 
for all of the codebase.  Over the last year, Greg, Benjamin, Mick, Josh, 
Scott, and I have been tracking down all of the contributors of the DataStax 
Java driver that had not signed the DataStax CLA and asked them to please sign 
that one or the ASF CLA.  After having discussed the CLAs with the PMC and ASF 
legal, we believe we are ready to proceed.

At this point, we'd like to propose CEP-8 for consideration, starting the 
process to accept the DataStax Java driver as an official ASF project.

https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-8%3A+Datastax+Drivers+Donation

Jeremy

Re: [DISCUSS] Bring cassandra-harry in tree as a submodule

[Jeremy Hanna]

I think it would be great to onboard Harry more officially into the project.  
However it would be nice to perhaps do some sanity checking outside of Apple 
folks to see how approachable it is.  That is, can someone take it and just run 
it with the current readme without any additional context?

I wonder if a mini-onboarding session would be good as a community session - go 
over Harry, how to run it, how to add a test?  Would that be the right venue?  
I just would like to see how we can not only plug it in to regular CI but get 
everyone that wants to add a test be able to know how to get started with it.

Jeremy

> On May 16, 2023, at 1:34 PM, Abe Ratnofsky  wrote:
> 
> Just to make sure I'm understanding the details, this would mean 
> apache/cassandra-harry maintains its status as a separate repository, 
> apache/cassandra references it as a submodule, and clones and builds Harry 
> locally, rather than pulling a released JAR. We can then reference Harry as a 
> library without maintaining public artifacts for it. Is that in line with 
> what you're thinking?
> 
> > I'd also like to see us get a Harry run integrated as part of our 
> > pre-commit CI
> 
> I'm a strong supporter of this, of course.
> 
>> On May 16, 2023, at 11:03 AM, Josh McKenzie  wrote:
>> 
>> Similar to what we've done with accord in 
>> https://issues.apache.org/jira/browse/CASSANDRA-18204, I'd like to discuss 
>> bringing cassandra-harry in-tree as a submodule. repo link: 
>> https://github.com/apache/cassandra-harry
>> 
>> Given the value it's brought to the project's stabilization efforts and the 
>> movement of other things in the ecosystem to being more integrated (accord, 
>> build-scripts https://issues.apache.org/jira/browse/CASSANDRA-18133), I 
>> think having the testing framework better localized and integrated would be 
>> a net benefit for adoption, awareness, maintenance, and tighter workflows as 
>> we troubleshoot future failures it surfaces.
>> 
>> I'd also like to see us get a Harry run integrated as part of our pre-commit 
>> CI (a 5 minute simple soak test for instance) and having that local in this 
>> fashion should make that a cleaner integration as well.
>> 
>> Thoughts?
> 

Re: CEP-30: Approximate Nearest Neighbor(ANN) Vector Search via Storage-Attached Indexes

[Jeremy Hanna]

Just wanted to add that I don't have any special knowledge of CEP-30 beyond 
what Jonathan posted and just trying to help clarify and answer questions as I 
can with some knowledge and experience from DSE Search and SAI.  Thanks to 
Caleb for helping validate some things as well.  And to be clear about partial 
results - the default with DSE Search at least is to fail a query if it can't 
get the full token range coverage.  However there is an option to allow for 
shards being unavailable and return partial results.

> On May 9, 2023, at 3:38 PM, Jeremy Hanna  wrote:
> 
> I talked to David and some others in slack to hopefully clarify:
> 
> With SAI, can you have partial results?  When you have a query that is 
> non-key based, you need to have full token range coverage of the results.  If 
> that isn't possible, will Vector Search/SAI return partial results?
> 
> Anything can happen in the implementation, but for scoring, it may not make 
> sense to return partial results because it's misleading.  For non-global 
> queries, it could or couldn't return partial results depending on 
> implementation/configuration.  In DSE you could have partial results 
> depending on the options.   However I couldn't find partial results defined 
> in CEP-7 or CEP-30.
> 
> The other questions are about scoring.
> 
> First, how is ordering/scoring done?
> 
> Each replica returns back to the coordinator a sorted set of results and the 
> coordinator will have to see all of the results globally in order to do a 
> global ordering.  You can't know what the top result is unless you've seen 
> everything.  As to the scoring, I'm not sure how that will get calculated.
> 
> Second, if I am ordering the results like for a Vector Search and I want to 
> have the top 1 result.  How is the scoring done and what happens if there are 
> 20 that have the same score?  How will the coordinator decide which 1 is 
> returned out of 20?
> 
> It returns results in token/partition and then clustering order.
> 
>> On May 9, 2023, at 2:53 PM, Caleb Rackliffe  wrote:
>> 
>> Anyone on this ML who still remembers DSE Search (or has experience w/ 
>> Elastic or SolrCloud) probably also knows that there are some significant 
>> pieces of an optimized scatter/gather apparatus for IR (even without 
>> sorting, which also doesn't exist yet) that do not exist in C* or it's range 
>> query system (which SAI and all other 2i implementations use). SAI, like all 
>> C* 2i implementations, is still a local index, and as that is the case, 
>> anything built on it will perform best in partition-scoped (at least on the 
>> read side) use-cases. (On the bright side, the project is moving toward 
>> larger partitions being a possibility.) With smaller clusters or use-cases 
>> that are extremely write-heavy/read-light, it's possible that the full 
>> scatter/gather won't be too onerous, especially w/ a few small tweaks (on 
>> top of a non-vnode cluster) to a.) keep fanout minimal and b.) keep 
>> range/index queries to a single pass to minimize latency.
>> 
>> Whatever we do, we just need to avoid a situation down the road where users 
>> don't understand these nuances and hit a wall where they try to use this in 
>> a way that is fundamentally incompatible w/ the way the database 
>> scales/works. (I've done my best to call this out in all discussions around 
>> SAI over time, and there may even end up being further guardrails put in 
>> place to make it even harder to misuse it...but I digress.)
>> 
>> Having said all that, I don't fundamentally have a problem w/ the proposal.
>> 
>> On Tue, May 9, 2023 at 2:11 PM Benedict > <mailto:bened...@apache.org>> wrote:
>>> HNSW can in principle be made into a distributed index. But that would be 
>>> quite a different paradigm to SAI.
>>> 
>>>> On 9 May 2023, at 19:30, Patrick McFadin >>> <mailto:pmcfa...@gmail.com>> wrote:
>>>> 
>>>> 
>>>> Under the goals section, there is this line:
>>>> 
>>>> Scatter/gather across replicas, combining topK from each to get global 
>>>> topK.
>>>> 
>>>> But what I'm hearing is, exactly how will that happen? Maybe this is an 
>>>> SAI question too. How is that verified in SAI?
>>>> 
>>>> On Tue, May 9, 2023 at 11:07 AM David Capwell >>> <mailto:dcapw...@apple.com>> wrote:
>>>>> Approach section doesn’t go over how this will handle cross replica 
>>>>> search, this would be good to flesh out… given results have a real 
>>>>> ranking, the current 2i logic may yield inco

Re: CEP-30: Approximate Nearest Neighbor(ANN) Vector Search via Storage-Attached Indexes

[Jeremy Hanna]

I talked to David and some others in slack to hopefully clarify:

With SAI, can you have partial results?  When you have a query that is non-key 
based, you need to have full token range coverage of the results.  If that 
isn't possible, will Vector Search/SAI return partial results?

Anything can happen in the implementation, but for scoring, it may not make 
sense to return partial results because it's misleading.  For non-global 
queries, it could or couldn't return partial results depending on 
implementation/configuration.  In DSE you could have partial results depending 
on the options.   However I couldn't find partial results defined in CEP-7 or 
CEP-30.

The other questions are about scoring.

First, how is ordering/scoring done?

Each replica returns back to the coordinator a sorted set of results and the 
coordinator will have to see all of the results globally in order to do a 
global ordering.  You can't know what the top result is unless you've seen 
everything.  As to the scoring, I'm not sure how that will get calculated.

Second, if I am ordering the results like for a Vector Search and I want to 
have the top 1 result.  How is the scoring done and what happens if there are 
20 that have the same score?  How will the coordinator decide which 1 is 
returned out of 20?

It returns results in token/partition and then clustering order.

> On May 9, 2023, at 2:53 PM, Caleb Rackliffe  wrote:
> 
> Anyone on this ML who still remembers DSE Search (or has experience w/ 
> Elastic or SolrCloud) probably also knows that there are some significant 
> pieces of an optimized scatter/gather apparatus for IR (even without sorting, 
> which also doesn't exist yet) that do not exist in C* or it's range query 
> system (which SAI and all other 2i implementations use). SAI, like all C* 2i 
> implementations, is still a local index, and as that is the case, anything 
> built on it will perform best in partition-scoped (at least on the read side) 
> use-cases. (On the bright side, the project is moving toward larger 
> partitions being a possibility.) With smaller clusters or use-cases that are 
> extremely write-heavy/read-light, it's possible that the full scatter/gather 
> won't be too onerous, especially w/ a few small tweaks (on top of a non-vnode 
> cluster) to a.) keep fanout minimal and b.) keep range/index queries to a 
> single pass to minimize latency.
> 
> Whatever we do, we just need to avoid a situation down the road where users 
> don't understand these nuances and hit a wall where they try to use this in a 
> way that is fundamentally incompatible w/ the way the database scales/works. 
> (I've done my best to call this out in all discussions around SAI over time, 
> and there may even end up being further guardrails put in place to make it 
> even harder to misuse it...but I digress.)
> 
> Having said all that, I don't fundamentally have a problem w/ the proposal.
> 
> On Tue, May 9, 2023 at 2:11 PM Benedict  > wrote:
>> HNSW can in principle be made into a distributed index. But that would be 
>> quite a different paradigm to SAI.
>> 
>>> On 9 May 2023, at 19:30, Patrick McFadin >> > wrote:
>>> 
>>> 
>>> Under the goals section, there is this line:
>>> 
>>> Scatter/gather across replicas, combining topK from each to get global topK.
>>> 
>>> But what I'm hearing is, exactly how will that happen? Maybe this is an SAI 
>>> question too. How is that verified in SAI?
>>> 
>>> On Tue, May 9, 2023 at 11:07 AM David Capwell >> > wrote:
 Approach section doesn’t go over how this will handle cross replica 
 search, this would be good to flesh out… given results have a real 
 ranking, the current 2i logic may yield incorrect results… so would think 
 we need num_ranges / rf queries in the best case, with some new capability 
 to sort the results?  If my assumption is correct, then how errors are 
 handled should also be fleshed out… Example: 1k cluster without vnode and 
 RF=3, so 333 queries fanned out to match, then coordinator needs to sort… 
 if 1 of the queries fails and can’t fall back to peers… does the query 
 fail (I assume so)?
 
> On May 8, 2023, at 7:20 PM, Jonathan Ellis  > wrote:
> 
> Hi all,
> 
> Following the recent discussion threads, I would like to propose CEP-30 
> to add Approximate Nearest Neighbor (ANN) Vector Search via 
> Storage-Attached Indexes (SAI) to Apache Cassandra.
> 
> The primary goal of this proposal is to implement ANN vector search 
> capabilities, making Cassandra more useful to AI developers and 
> organizations managing large datasets that can benefit from fast 
> similarity search.
> 
> The implementation will leverage Lucene's Hierarchical Navigable Small 
> World (HNSW) library and introduce a new CQL data type for vector 
> embeddings, a new SAI 

Re: [VOTE] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[Jeremy Hanna]

+1 nb, I had to run Cassandra + Hadoop from the early days (0.7+) and it was 
painful.  This is a major step forward.

> On May 4, 2023, at 1:44 PM, Patrick McFadin  wrote:
> 
> As somebody who gave this talk: https://youtu.be/9xf_IXNylhM I love the 
> evolution of this topic. Excited to see this! ++1 nb
> 
> Patrick
> 
> 
> 
> On Thu, May 4, 2023 at 11:35 AM C. Scott Andreas  > wrote:
>> +1nb.
>> 
>> As someone familiar with this work, it's pretty hard to overstate the impact 
>> it has on completing Cassandra's HTAP story. Eliminating the overhead of 
>> bulk reads and writes on production OLTP clusters is transformative.
>> 
>> – Scott
>> 
>>> On May 4, 2023, at 9:47 AM, Doug Rohrer >> > wrote:
>>> 
>>> 
>>> Hello all,
>>> 
>>> I’d like to put CEP-28 to a vote.
>>> 
>>> Proposal:
>>> 
>>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-28%3A+Reading+and+Writing+Cassandra+Data+with+Spark+Bulk+Analytics
>>> 
>>> Jira:
>>> https://issues.apache.org/jira/browse/CASSANDRA-16222
>>> 
>>> Draft implementation:
>>> 
>>> - Apache Cassandra Spark Analytics source code: 
>>> https://github.com/frankgh/cassandra-analytics
>>> - Changes required for Sidecar: 
>>> https://github.com/frankgh/cassandra-sidecar/tree/CEP-28-bulk-apis
>>> 
>>> Discussion:
>>> https://lists.apache.org/thread/lrww4d7cdxgtg8o3gt8b8foymzpvq7z3
>>> 
>>> The vote will be open for 72 hours. 
>>> A vote passes if there are at least three binding +1s and no binding 
>>> vetoes. 
>>> 
>>> 
>>> Thanks,
>>> 
>>> Doug Rohrer
>>> 
>>> 
>>> 
>> 
>> 

Re: [POLL] Vector type for ML

[Jeremy Hanna]

I'm all for bringing more functionality to the masses sooner, but the original 
idea has a very very specific use case.  Do we have use cases for a general 
purpose Vector/Array data structure?  If so, awesome.  I just wondered if 
generalizing provides value, beyond being straightforward to implement.  I'm 
just trying to be sensitive to the database code maintenance and driver support 
for general types versus a single type for a specific, well defined purpose.

If it could easily be a plugin, that's great - but the full picture involves 
drivers that need to support it or you end up getting binary blobs you have to 
decode client side and then do stuff with.  So ideally if you have a well 
defined use case that you can build into the database, having it just be part 
of the database and associated drivers - that makes the experience much much 
better.

I'm not trying to say B couldn't be valuable or that a plugin couldn't be 
feasible.  I'm just trying to enlarge the picture a bit to see what that means 
for this use case and for the supporting drivers/clients.

> On May 2, 2023, at 3:04 PM, Benedict  wrote:
> 
> But it’s so trivial it was already implemented by David in the span of ten 
> minutes? If anything, we’re slowing progress down by refusing to do the extra 
> types, as we’re busy arguing about it rather than delivering a feature?
> 
> FWIW, my interpretation of the votes today is that we SHOULD NOT (ever) 
> support types beyond float. Not that we should start with float.
> 
> So, this whole debate is a mess, I think. But hey ho.
> 
>> On 2 May 2023, at 20:57, Patrick McFadin  wrote:
>> 
>> 
>> I'll speak up on that one. If you look at my ranked voting, that is where my 
>> head is. I get accused of scope creep (a lot) and looking at the initial 
>> proposal Jonathan put on the ML it was mostly "Developers are adopting 
>> vector search at a furious pace and I think I have a simple way of adding 
>> support to keep Cassandra relevant for these use cases" Instead of just 
>> focusing on this use case, I feel the arguments have bike shedded into scope 
>> creep which means it will take forever to get into the project.
>> 
>> My preference is to see one thing validated with an MVP and get it into the 
>> hands of developers sooner so we can continue to iterate based on actual 
>> usage. 
>> 
>> It doesn't say your points are wrong or your opinions are broken, I'm voting 
>> for what I think will be awesome for users sooner. 
>> 
>> Patrick
>> 
>> On Tue, May 2, 2023 at 12:29 PM Benedict > > wrote:
>>> Could folk voting against a general purpose type (that could well be called 
>>> a vector) briefly explain their reasoning?
>>> 
>>> We established in the other thread that it’s technically trivial, meaning 
>>> folk must think it is strictly superior to only support float rather than 
>>> eg all numeric types (note: for the type, not the ANN). 
>>> 
>>> I am surprised, and the blurbs accompanying votes so far don’t seem to 
>>> touch on this, mostly just endorsing the idea of a vector.
>>> 
>>> 
 On 2 May 2023, at 20:20, Patrick McFadin >>> > wrote:
 
 
 A > B > C on both polls. 
 
 Having talked to several users in the community that are highly excited 
 about this change, this gets to what developers want to do at Cassandra 
 scale: store embeddings and retrieve them. 
 
 On Tue, May 2, 2023 at 11:47 AM Andrés de la Peña >>> > wrote:
> A > B > C
> 
> I don't think that ML is such a niche application that it can't have its 
> own CQL data type. Also, vectors are mathematical elements that have more 
> applications that ML.
> 
> On Tue, 2 May 2023 at 19:15, Mick Semb Wever  > wrote:
>> 
>> 
>> On Tue, 2 May 2023 at 17:14, Jonathan Ellis > > wrote:
>>> Should we add a vector type to Cassandra designed to meet the needs of 
>>> machine learning use cases, specifically feature and embedding vectors 
>>> for training, inference, and vector search?  
>>> 
>>> ML vectors are fixed-dimension (fixed-length) sequences of numeric 
>>> types, with no nulls allowed, and with no need for random access. The 
>>> ML industry overwhelmingly uses float32 vectors, to the point that the 
>>> industry-leading special-purpose vector database ONLY supports that 
>>> data type.
>>> 
>>> This poll is to gauge consensus subsequent to the recent discussion 
>>> thread at 
>>> https://lists.apache.org/thread/0lj1nk9jbhkf1rlgqcvxqzfyntdjrnk0.
>>> 
>>> Please rank the discussed options from most preferred option to least, 
>>> e.g., A > B > C (A is my preference, followed by B, followed by C) or C 
>>> > B = A (C is my preference, followed by B or A approximately equally.)
>>> 
>>> (A) I am in favor of adding a vector type for 

Re: [DISCUSS] CEP-29 CQL NOT Operator

[Jeremy Hanna]

It can be from not specifying the full primary key including clustering columns or it can be across multiple partitions. There are two scenarios. That’s why I created https://issues.apache.org/jira/browse/CASSANDRA-15803 and why I think it’s relevant for this. On Apr 14, 2023, at 9:37 AM, Lorina Poland  wrote:Wow, you know, J.D., I've never actually heard ALLOW FILTERING described as you did. Generally, the discussion is always in terms of multiple partitions, probably because that is the situation in which the memory is exceeded. Thanks for that definition. Regardless of how this discussion goes, I'll make a ticket to change that doc.LorinaOn Thu, Apr 13, 2023 at 4:17 AM J. D. Jordan  wrote:The documentation is wrong. ALLOW FILTERING has always meant that “rows will need to be materialized in memory and accepted or rejected by a column filter” aka the full primary key was not specified and some other column was specified.  It has never been about multiple partitions.Basically “will the server need to read from disk more data (possibly a lot more) than will be returned to the client”.Should we change how that works? Maybe. But let move such discussions to a new thread and keep this one about the CEP proposal.On Apr 13, 2023, at 6:00 AM, Andrés de la Peña  wrote:Indeed requiring AF for "select * from ks.tb where p1 = 1 and c1 = 2 and col2 = 1", where p1 and c1 are all the columns in the primary key, sounds like a bug. I think the criterion in the code is that we require AF if there is any column restriction that cannot be processed by the primary key or a secondary index. The error message indeed seems to reject any kind of filtering, independently of primary key filters. We can see this even without defined clustering keys:CREATE TABLE t (k int PRIMARY KEY, v int);SELECT * FROM  t WHERE  k = 1 AND v = 1; # requires AFThat clashes with documentation, where it's said that AF is required for filters that require scanning all partitions. If we were to adapt the code to the behaviour described in documentation we shouldn't require AF if there are restrictions specifying a partition key. Or possibly a group of partition keys, if a IN restriction is used. So both within row and within partition filtering wouldn't require AF.Regarding adding a new ALLOW FILTERING WITHIN PARTITION, I think we could just add a guardrail to directly disallow those queries, without needing to add the WITHIN PARTITION clause to the CQL grammar.On Thu, 13 Apr 2023 at 11:11, Henrik Ingo  wrote:On Thu, Apr 13, 2023 at 10:20 AM Miklosovic, Stefan  wrote:Somebody correct me if I am wrong but "partition key" itself is not enough (primary keys = partition keys + clustering columns). It will require ALLOW FILTERING when clustering columns are not specified either.

create table ks.tb (p1 int, c1 int, col1 int, col2 int, primary key (p1, c1));
select * from ks.tb where p1 = 1 and col1 = 2;     // this will require allow filtering

The documentation seems to omit this fact.It does seem so.That said, personally I was assuming, and would still argue it's the optimal choice, that the documentation was right and reality is wrong.If there is a partition key, then the query can avoid scanning the entire table, across all nodes, potentially petabytes.If a query specifies a partition key but not the full clustering key, of course there will be some scanning needed, but this is marginal compared to the need to scan the entire table. Even in the worst case, a partition with 2 billion cells, we are talking about seconds to filter the result from the single partition.> Aha I get what you all mean:No, I actually think both are unnecessary. But yeah, certainly this latter case is a bug?henrik-- Henrik Ingoc. +358 40 569 7354 w. www.datastax.com      

Re: [DISCUSS] CEP-29 CQL NOT Operator

[Jeremy Hanna]

Considering all of the examples require using ALLOW FILTERING with the 
partition key specified, I think it's appropriate to consider separating out 
use of ALLOW FILTERING within a partition versus ALLOW FILTERING across the 
whole table.  A few years back we had a discussion about this in ASF slack in 
the context of capability restrictions and it seems relevant here.  That is, we 
don't want people to get comfortable using ALLOW FILTERING across the whole 
table.  However, there are times when ALLOW FILTERING within a partition is 
reasonable.

Ticket to discuss separating them out: 
https://issues.apache.org/jira/browse/CASSANDRA-15803
Summary: Perhaps add an optional [WITHIN PARTITION] or something similar to 
make it backwards compatible and indicate that this is purely within the 
specified partition.

This also gives us the ability to disallow table scan types of ALLOW FILTERING 
from a guard rail perspective, because the intent is explicit.  That operators 
could disallow ALLOW FILTERING but allow ALLOW FILTERING WITHIN PARTITION, or 
whatever is decided.

I do NOT want to hijack a good discussion but I thought this separation could 
be useful within this context.

Jeremy

> On Apr 6, 2023, at 3:00 PM, Patrick McFadin  wrote:
> 
> I love that this is finally coming to Cassandra. Absolutely hate that, once 
> again, we'll be endorsing the use of ALLOW FILTERING. This is an anti-pattern 
> that keeps getting legitimized.
> 
> Hot take: Should we just not do Milestones 1 and 2 and wait for an index-only 
> Milestone 3? 
> 
> Patrick
> 
> On Thu, Apr 6, 2023 at 10:04 AM David Capwell  > wrote:
>> Overall I welcome this feature, was trying to use this around 1-2 months 
>> back and found we didn’t support, so glad to see it coming!
>> 
>> From a testing point of view, I think we would want to have good fuzz 
>> testing covering complex types (frozen/non-frozen collections, tuples, udt, 
>> etc.), and reverse ordering; both sections tend to cause the most problem 
>> for new features (and existing ones)
>> 
>> We also will want a way to disable this feature, and optionally disable at 
>> different sections (such as m2’s NOT IN for partition keys).
>> 
>> > On Apr 4, 2023, at 2:28 AM, Piotr Kołaczkowski > > > wrote:
>> > 
>> > Hi everyone!
>> > 
>> > I created a new CEP for adding NOT support to the query language and
>> > want to start discussion around it:
>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-29%3A+CQL+NOT+operator
>> > 
>> > Happy to get your feedback.
>> > --
>> > Piotr
>> 

Re: [VOTE] CEP-26: Unified Compaction Strategy

[Jeremy Hanna]

+1 nb, will be great to have this in the codebase - it will make nearly every 
table's compaction work more efficiently.  The only possible exception is 
tables that are well suited for TWCS.

> On Apr 4, 2023, at 8:00 AM, Berenguer Blasi  wrote:
> 
> +1
> 
> On 4/4/23 14:36, J. D. Jordan wrote:
>> +1
>> 
>>> On Apr 4, 2023, at 7:29 AM, Brandon Williams  
>>>  wrote:
>>> 
>>> 
>>> +1
>>> 
>>> On Tue, Apr 4, 2023, 7:24 AM Branimir Lambov >> > wrote:
 Hi everyone,
 
 I would like to put CEP-26 to a vote.
 
 Proposal:
 https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-26%3A+Unified+Compaction+Strategy
 
 JIRA and draft implementation:
 https://issues.apache.org/jira/browse/CASSANDRA-18397
 
 Up-to-date documentation:
 https://github.com/blambov/cassandra/blob/CASSANDRA-18397/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md
 
 Discussion:
 https://lists.apache.org/thread/8xf5245tclf1mb18055px47b982rdg4b
 
 The vote will be open for 72 hours.
 A vote passes if there are at least three binding +1s and no binding 
 vetoes.
 
 Thanks,
 Branimir

Re: [DISCUSS] CEP-28: Reading and Writing Cassandra Data with Spark Bulk Analytics

[Jeremy Hanna]

Thank you for the write-up and the efforts on CASSANDRA-16222.  It sounds like 
you've been using this for some time.  I understand from the rejected 
alternatives that the Spark Cassandra Connector was slower because it goes 
through the read and write path for C* rather than this backdoor mechanism.  In 
your experience using this, under what circumstances have you seen that this 
tool is not a good fit for analytics - such as complex predicates?  The 
challenge with the Spark Cassandra Connector and previously the Hadoop 
integration is that it had to do full table scans even to get small amounts of 
data.  It sounds like this is similar in that it has to do a full table scan 
but with the advantage of being faster and less load on the cluster.  In other 
words, I'm asking if this has been a replacement for the Spark Cassandra 
Connector or if there are cases in your work where SCC is a better fit.

Also to Benjamin's point in the comments on the CEP itself, how coupled is this 
to internals?  Are there going to be higher level APIs or is it going to call 
internal storage classes directly?

Thanks!

Jeremy


> On Mar 23, 2023, at 12:33 PM, Doug Rohrer  wrote:
> 
> Hi everyone,
> 
> Wiki: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-28%3A+Reading+and+Writing+Cassandra+Data+with+Spark+Bulk+Analytics
> 
> We’d like to propose this CEP for adoption by the community.
> 
> It is common for teams using Cassandra to find themselves looking for a way 
> to interact with large amounts of data for analytics workloads. However, 
> Cassandra’s standard APIs aren’t designed for large scale data egress/ingest 
> as the native read/write paths weren’t designed for bulk analytics.
> 
> We’re proposing this CEP for this exact purpose. It enables the 
> implementation of custom Spark (or similar) applications that can either read 
> or write large amounts of Cassandra data at line rates, by accessing the 
> persistent storage of nodes in the cluster via the Cassandra Sidecar.
> 
> This CEP proposes new APIs in the Cassandra Sidecar and a companion library 
> that allows deep integration into Apache Spark that allows its users to bulk 
> import or export data from a running Cassandra cluster with minimal to no 
> impact to the read/write traffic.
> 
> We will shortly publish a branch with code that will accompany this CEP to 
> help readers understand it better.
> 
> As a reminder, please keep the discussion here on the dev list vs. in the 
> wiki, as we’ve found it easier to manage via email.
> 
> Sincerely,
> 
> Doug Rohrer & James Berragan

Re: Welcome our next PMC Chair Josh McKenzie

[Jeremy Hanna]

Thank you Mick for all of your hard work in the project including your time as 
the PMC chair!

Thank you too Josh for all that you do - and for the work that you'll do as 
chair!

> On Mar 23, 2023, at 3:22 AM, Mick Semb Wever  wrote:
> 
> It is time to pass the baton on, and on behalf of the Apache Cassandra 
> Project Management Committee (PMC) I would like to welcome and congratulate 
> our next PMC Chair Josh McKenzie (jmckenzie).
> 
> Most of you already know Josh, especially through his regular and valuable 
> project oversight and status emails, always presenting a balance and 
> understanding to the various views and concerns incoming. 
> 
> Repeating Paulo's words from last year: The chair is an administrative 
> position that interfaces with the Apache Software Foundation Board, by 
> submitting regular reports about project status and health. Read more about 
> the PMC chair role on Apache projects:
> - https://www.apache.org/foundation/how-it-works.html#pmc
> - https://www.apache.org/foundation/how-it-works.html#pmc-chair
> - https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers
> 
> The PMC as a whole is the entity that oversees and leads the project and any 
> PMC member can be approached as a representative of the committee. A list of 
> Apache Cassandra PMC members can be found on: 
> https://cassandra.apache.org/_/community.html

Re: [DISCUSS] CEP-26: Unified Compaction Strategy

[Jeremy Hanna]

You're right that it doesn't handle it in the sense that it doesn't resolve it 
the problem, but it also doesn't do what STCS does.  From what I've seen, STCS 
blindly tries to compact and then the disk will fill up triggering the disk 
failure policy.  With UCS it's much less likely and if it does happen, my 
understanding is that it will skip the compaction.  I didn't realize that LCS 
would try to reduce the scope of the compaction.  I can't find in the branch 
where it handles that.

Branimir, can you point to where it handles the scenario?

Thanks,

Jeremy

> On Mar 17, 2023, at 4:52 PM, Jeff Jirsa  wrote:
> 
> 
> 
>> On Mar 17, 2023, at 1:46 PM, Jeremy Hanna  wrote:
>> 
>> 
>> 
>> One much more graceful element of UCS is that instead of what was previously 
>> done with compaction strategies where the server just shuts down when 
>> running out of space - forcing system administrators to be paranoid about 
>> headroom.  Instead UCS has a target overhead (default 20%).  First since the 
>> ranges are sharded, it makes it less likely that there will be large 
>> sstables that need to get compacted to require as much headroom, but  if it 
>> detects that there is a compaction that will violate the target overhead, it 
>> will log that and skip the compaction - a much more graceful way of handling 
>> it.
> 
> Skipping doesn’t really handle it though? 
> 
> If you have a newly flushed sstable full of tombstones and it naturally 
> somehow triggers you to exceed that target overhead you never free that 
> space? Usually LCS would try to reduce the scope of the compaction, and I 
> assume UCS will too? 
> 
> 

Re: [DISCUSS] CEP-26: Unified Compaction Strategy

[Jeremy Hanna]

I think it's important to highlight a few of practical elements of this work 
that may not be completely clear.

Sharding ranges allows for parallel compactions - taking advantage of 
additional CPUs on the server.  This allows compactions to keep up much easier.

One much more graceful element of UCS is that instead of what was previously 
done with compaction strategies where the server just shuts down when running 
out of space - forcing system administrators to be paranoid about headroom.  
Instead UCS has a target overhead (default 20%).  First since the ranges are 
sharded, it makes it less likely that there will be large sstables that need to 
get compacted to require as much headroom, but  if it detects that there is a 
compaction that will violate the target overhead, it will log that and skip the 
compaction - a much more graceful way of handling it.

One other useful practical element is the leveling that Branimir mentions.  The 
scaling_parameters (previously static_scaling_factors) represent what to do at 
each level.  They represent how and when to compact and go from an extreme of 
aggressive compacting to reduce read amplification to the other extreme of 
leaving sstables alone to reduce write amplification.  While you can set it to 
a single value, that will be the value for all levels.  However, making that 
value different for each level has some benefits.  For example, in lower levels 
you can have it compact more aggressively and then as it levels up, you could 
have it gradually reduce compactions.

Another practical element is that if you want to change settings or if the 
strategy automatically changes settings based on load (I believe that's where 
Branimir is heading with this), the strategy will start using the new settings 
without having to recompact existing data.

Finally to Benedict's point about density, unified compaction even with the 
default parameters has been shown to scale to many TB of data per node while 
keeping latencies low.

I'm excited for this to get incorporated into the project and improved.

> On Mar 17, 2023, at 11:45 AM, Josh McKenzie  wrote:
> 
> Could we get a JIRA for this too so we can get some reviewers collaborating 
> on this? Only see Lorina's ticket for documenting it in JIRA atm.
> 
> On Fri, Mar 17, 2023, at 9:53 AM, Branimir Lambov wrote:
>> The prototype of UCS can now be found in this pull request: 
>> https://github.com/apache/cassandra/pull/2228
>> 
>> Its description is given in the included markdown documentation: 
>> https://github.com/blambov/cassandra/blob/UCS-density/src/java/org/apache/cassandra/db/compaction/UnifiedCompactionStrategy.md
>> 
>> The latest code includes some new elements compared to the link Henrik 
>> posted, including density levelling, bucketing based solely on overlap, and 
>> output splitting by expected density. It goes a little further than what is 
>> described in the CEP-26 proposal as prototyping showed that we can make the 
>> selection of sstables to compact and the sharding decisions independent of 
>> each other. This makes the strategy more stable and better able to react to 
>> changes in configuration and environment.
>> 
>> Regards,
>> Branimir
>> 
>> On Wed, Dec 21, 2022 at 10:01 AM Benedict > > wrote:
>> 
>> I’m personally very excited by this work. Compaction could do with a spring 
>> clean and this feels to formalise things much more cleanly, but density 
>> tiering in particular is something I’ve wanted to incorporate for years now, 
>> as it should significantly improve STCS behaviour (most importantly reducing 
>> read amplification and the amount of disk space required, narrowing the 
>> performance delta to LCS in these important dimensions), and simplifies 
>> re-levelling of LCS, making large streams much less painful.
>> 
>> 
>>> On 21 Dec 2022, at 07:19, Henrik Ingo >> > wrote:
>>> 
>>> I noticed the CEP doesn't link to this, so it should be worth mentioning 
>>> that the UCS documentation is available here: 
>>> https://github.com/datastax/cassandra/blob/ds-trunk/doc/unified_compaction.md
>>> 
>>> Both of the above seem to do a poor job referencing the literature we've 
>>> been inspired by. I will link to Mark Callaghan's blog on the subject:
>>> 
>>> http://smalldatum.blogspot.com/2018/07/tiered-or-leveled-compaction-why-not.html?m=1
>>>  
>>> 
>>> 
>>> ...and lazily will also borrow from Mark a post that references a bunch of 
>>> LSM (not just UCS related) academic papers: 
>>> http://smalldatum.blogspot.com/2018/08/name-that-compaction-algorithm.html?m=1
>>>  
>>> 

Re: Role of Hadoop code in Cassandra 5.0

[Jeremy Hanna]

Regarding deprecation, while I support the deprecation and removal from the 
Cassandra codebase, I do think we should communicate that with the wider 
community (user thread?) so people aren't surprised - especially since it's 
already four months after the 4.1.0 release.  That would hopefully also 
encourage those interested in continuing support to extract it out into a 
separate library.

> On Mar 16, 2023, at 4:19 PM, Miklosovic, Stefan 
>  wrote:
> 
> I think we already decided it in this thread.
> 
> I was specifically asking this question:
> 
> Deprecation would mean that the code has to be there whole 5.0 so we can 
> remove it for real in 6.0?
> 
> To which the response was:
> 
> I think if we reach consensus here that decides it. I too vote to
> deprecate in 4.1.x.  This means we would remove it in 5.0.
> 
> Then bunch of +1s followed and agreed with that explicitly.
> 
> I do not plan to maintain nor extract that, personally.
> 
> 
> From: David Capwell mailto:dcapw...@apple.com>>
> Sent: Thursday, March 16, 2023 22:13
> To: dev@cassandra.apache.org <mailto:dev@cassandra.apache.org>
> Subject: Re: Role of Hadoop code in Cassandra 5.0
> 
> NetApp Security WARNING: This is an external email. Do not click links or 
> open attachments unless you recognize the sender and know the content is safe.
> 
> 
> 
> Isn’t our deprecation rules that if we deprecate in 4.0.0 we can remove in 
> 5.x, but 4.x needs to wait for 6.x?  I am cool deprecating this and willing 
> to pull into another repo if people (not me) are willing to maintain it (else 
> just delete).
> 
> On Mar 10, 2023, at 1:13 AM, Jacek Lewandowski  
> wrote:
> 
> I've experimentally added 
> https://issues.apache.org/jira/browse/CASSANDRA-16984 to 
> https://issues.apache.org/jira/browse/CASSANDRA-18306 (post 4.0 cleanup)
> 
> - - -- --- -  -
> Jacek Lewandowski
> 
> 
> pt., 10 mar 2023 o 09:56 Berenguer Blasi  <mailto:berenguerbl...@gmail.com><mailto:berenguerbl...@gmail.com>> 
> napisał(a):
> 
> +1 deprecate + removal
> 
> On 10/3/23 1:41, Jeremy Hanna wrote:
> It was mainly to integrate with Hadoop - I used it from 0.6 to 1.2 in 
> production prior to starting at DataStax and at that time I was stitching 
> together Cloudera's distribution of Hadoop with Cassandra.  Back then there 
> were others that used it as well.  As far as I know, usage dropped off when 
> the Spark Cassandra Connector got pretty mature.  It enabled people to take 
> an off the shelf Hadoop distribution and run the Hadoop processes on the same 
> nodes or external to the Cassandra cluster and get topology information to do 
> things like Hadoop splits and things like that through the Hadoop interfaces. 
>  I think the version lag is an indication that it hasn't been used recently.  
> Also, like others have said, the Spark Cassandra Connector is really what 
> people should be using at this point imo.  That or depending on the use case, 
> Apple's bulk reader: https://github.com/jberragan/spark-cassandra-bulkreader 
> that is mentioned on https://issues.apache.org/jira/browse/CASSANDRA-16222.
> 
> On Mar 9, 2023, at 12:00 PM, Rahul Xavier Singh  <mailto:rahul.xavier.si...@gmail.com>><mailto:rahul.xavier.si...@gmail.com> 
> wrote:
> 
> What is the hadoop code for? For interacting from Hadoop via CQL, or Thrift 
> if it's that old, or directly looking at SSTables? Been using C* since 2 and 
> have never used it.
> 
> Agree to deprecate in next possible 4.1.x version and remove in 5.0
> 
> Rahul Singh
> Chief Executive Officer | Business Platform Architect m: 202.905.2818 e: 
> rahul.si...@anant.us 
> <mailto:rahul.si...@anant.us><mailto:rahul.si...@anant.us> li: 
> http://linkedin.com/in/xingh ca: http://calendly.com/xingh
> 
> We create, support, and manage real-time global data & analytics platforms 
> for the modern enterprise.
> 
> Anant | https://anant.us<https://anant.us/>
> 3 Washington Circle, Suite 301
> Washington, D.C. 20037
> 
> http://Cassandra.Link<http://cassandra.link/> 
> <http://cassandra.link<http://cassandra.link/>> : The best resources for 
> Apache Cassandra
> 
> 
> On Thu, Mar 9, 2023 at 12:53 PM Brandon Williams  <mailto:dri...@gmail.com><mailto:dri...@gmail.com>> wrote:
> I think if we reach consensus here that decides it. I too vote to
> deprecate in 4.1.x.  This means we would remove it in 5.0.
> 
> Kind Regards,
> Brandon
> 
> On Thu, Mar 9, 2023 at 11:32 AM Ekaterina Dimitrova
>  <mailto:e.dimitr...@gmail.com><mailto:e.dimitr...@gmail.com>> wrot

Re: Role of Hadoop code in Cassandra 5.0

[Jeremy Hanna]

It was mainly to integrate with Hadoop - I used it from 0.6 to 1.2 in 
production prior to starting at DataStax and at that time I was stitching 
together Cloudera's distribution of Hadoop with Cassandra.  Back then there 
were others that used it as well.  As far as I know, usage dropped off when the 
Spark Cassandra Connector got pretty mature.  It enabled people to take an off 
the shelf Hadoop distribution and run the Hadoop processes on the same nodes or 
external to the Cassandra cluster and get topology information to do things 
like Hadoop splits and things like that through the Hadoop interfaces.  I think 
the version lag is an indication that it hasn't been used recently.  Also, like 
others have said, the Spark Cassandra Connector is really what people should be 
using at this point imo.  That or depending on the use case, Apple's bulk 
reader: https://github.com/jberragan/spark-cassandra-bulkreader that is 
mentioned on https://issues.apache.org/jira/browse/CASSANDRA-16222.

> On Mar 9, 2023, at 12:00 PM, Rahul Xavier Singh 
>  wrote:
> 
> What is the hadoop code for? For interacting from Hadoop via CQL, or Thrift 
> if it's that old, or directly looking at SSTables? Been using C* since 2 and 
> have never used it. 
> 
> Agree to deprecate in next possible 4.1.x version and remove in 5.0 
> 
> Rahul Singh
> Chief Executive Officer | Business Platform Architect
> m: 202.905.2818 e: rahul.si...@anant.us  li: 
> http://linkedin.com/in/xingh ca: http://calendly.com/xingh
> 
> We create, support, and manage real-time global data & analytics platforms 
> for the modern enterprise.
> 
> Anant | https://anant.us 
> 3 Washington Circle, Suite 301
> Washington, D.C. 20037
> 
> http://Cassandra.Link  : The best resources for 
> Apache Cassandra
> 
> 
> On Thu, Mar 9, 2023 at 12:53 PM Brandon Williams  > wrote:
>> I think if we reach consensus here that decides it. I too vote to
>> deprecate in 4.1.x.  This means we would remove it in 5.0.
>> 
>> Kind Regards,
>> Brandon
>> 
>> On Thu, Mar 9, 2023 at 11:32 AM Ekaterina Dimitrova
>> mailto:e.dimitr...@gmail.com>> wrote:
>> >
>> > Deprecation sounds good to me, but I am not completely sure in which 
>> > version we can do it. If it is possible to add a deprecation warning in 
>> > the 4.x series or at least 4.1.x - I vote for that.
>> >
>> > On Thu, 9 Mar 2023 at 12:14, Jacek Lewandowski 
>> > mailto:lewandowski.ja...@gmail.com>> wrote:
>> >>
>> >> Is it possible to deprecate it in the 4.1.x patch release? :)
>> >>
>> >>
>> >> - - -- --- -  -
>> >> Jacek Lewandowski
>> >>
>> >>
>> >> czw., 9 mar 2023 o 18:11 Brandon Williams > >> > napisał(a):
>> >>>
>> >>> This is my feeling too, but I think we should accomplish this by
>> >>> deprecating it first.  I don't expect anything will change after the
>> >>> deprecation period.
>> >>>
>> >>> Kind Regards,
>> >>> Brandon
>> >>>
>> >>> On Thu, Mar 9, 2023 at 11:09 AM Jacek Lewandowski
>> >>> mailto:lewandowski.ja...@gmail.com>> wrote:
>> >>> >
>> >>> > I vote for removing it entirely.
>> >>> >
>> >>> > thanks
>> >>> > - - -- --- -  -
>> >>> > Jacek Lewandowski
>> >>> >
>> >>> >
>> >>> > czw., 9 mar 2023 o 18:07 Miklosovic, Stefan 
>> >>> > mailto:stefan.mikloso...@netapp.com>> 
>> >>> > napisał(a):
>> >>> >>
>> >>> >> Derek,
>> >>> >>
>> >>> >> I have couple more points ... I do not think that extracting it to a 
>> >>> >> separate repository is "win". That code is on Hadoop 1.0.3. We would 
>> >>> >> be spending a lot of work on extracting it just to extract 10 years 
>> >>> >> old code with occasional updates (in my humble opinion just to make 
>> >>> >> it compilable again if the code around changes). What good is in 
>> >>> >> that? We would have one more place to take care of ... Now we at 
>> >>> >> least have it all in one place.
>> >>> >>
>> >>> >> I believe we have four options:
>> >>> >>
>> >>> >> 1) leave it there so it will be like this is for next years with 
>> >>> >> questionable and diminishing usage
>> >>> >> 2) update it to Hadoop 3.3 (I wonder who is going to do that)
>> >>> >> 3) 2) and extract it to a separate repository but if we do 2) we can 
>> >>> >> just leave it there
>> >>> >> 4) remove it
>> >>> >>
>> >>> >> 
>> >>> >> From: Derek Chen-Becker > >>> >> >
>> >>> >> Sent: Thursday, March 9, 2023 15:55
>> >>> >> To: dev@cassandra.apache.org 
>> >>> >> Subject: Re: Role of Hadoop code in Cassandra 5.0
>> >>> >>
>> >>> >> NetApp Security WARNING: This is an external email. Do not click 
>> >>> >> links or open attachments unless you recognize the sender and know 
>> >>> >> the content is safe.
>> >>> >>
>> >>> >>
>> >>> >>
>> >>> >> I think the question isn't "Who ... is still using that?" but more 
>> >>> >> "are we 

Re: Role of Hadoop code in Cassandra 5.0

[Jeremy Hanna]

+1 from me to deprecate in 4.x and remove in 5.0.

> On Mar 9, 2023, at 12:01 PM, J. D. Jordan  wrote:
> 
> +1 from me to deprecate in 4.x and remove in 5.0.
> 
> -Jeremiah
> 
>> On Mar 9, 2023, at 11:53 AM, Brandon Williams  wrote:
>> 
>> I think if we reach consensus here that decides it. I too vote to
>> deprecate in 4.1.x.  This means we would remove it in 5.0.
>> 
>> Kind Regards,
>> Brandon
>> 
>>> On Thu, Mar 9, 2023 at 11:32 AM Ekaterina Dimitrova
>>>  wrote:
>>> 
>>> Deprecation sounds good to me, but I am not completely sure in which 
>>> version we can do it. If it is possible to add a deprecation warning in the 
>>> 4.x series or at least 4.1.x - I vote for that.
>>> 
 On Thu, 9 Mar 2023 at 12:14, Jacek Lewandowski 
  wrote:
 
 Is it possible to deprecate it in the 4.1.x patch release? :)
 
 
 - - -- --- -  -
 Jacek Lewandowski
 
 
 czw., 9 mar 2023 o 18:11 Brandon Williams  napisał(a):
> 
> This is my feeling too, but I think we should accomplish this by
> deprecating it first.  I don't expect anything will change after the
> deprecation period.
> 
> Kind Regards,
> Brandon
> 
> On Thu, Mar 9, 2023 at 11:09 AM Jacek Lewandowski
>  wrote:
>> 
>> I vote for removing it entirely.
>> 
>> thanks
>> - - -- --- -  -
>> Jacek Lewandowski
>> 
>> 
>> czw., 9 mar 2023 o 18:07 Miklosovic, Stefan 
>>  napisał(a):
>>> 
>>> Derek,
>>> 
>>> I have couple more points ... I do not think that extracting it to a 
>>> separate repository is "win". That code is on Hadoop 1.0.3. We would be 
>>> spending a lot of work on extracting it just to extract 10 years old 
>>> code with occasional updates (in my humble opinion just to make it 
>>> compilable again if the code around changes). What good is in that? We 
>>> would have one more place to take care of ... Now we at least have it 
>>> all in one place.
>>> 
>>> I believe we have four options:
>>> 
>>> 1) leave it there so it will be like this is for next years with 
>>> questionable and diminishing usage
>>> 2) update it to Hadoop 3.3 (I wonder who is going to do that)
>>> 3) 2) and extract it to a separate repository but if we do 2) we can 
>>> just leave it there
>>> 4) remove it
>>> 
>>> 
>>> From: Derek Chen-Becker 
>>> Sent: Thursday, March 9, 2023 15:55
>>> To: dev@cassandra.apache.org
>>> Subject: Re: Role of Hadoop code in Cassandra 5.0
>>> 
>>> NetApp Security WARNING: This is an external email. Do not click links 
>>> or open attachments unless you recognize the sender and know the 
>>> content is safe.
>>> 
>>> 
>>> 
>>> I think the question isn't "Who ... is still using that?" but more "are 
>>> we actually going to support it?" If we're on a version that old it 
>>> would appear that we've basically abandoned it, although there do 
>>> appear to have been refactoring (for other things) commits in the last 
>>> couple of years. I would be in favor of removal from 5.0, but at the 
>>> very least, could it be moved into a separate repo/package so that it's 
>>> not pulling a relatively large dependency subtree from Hadoop into our 
>>> main codebase?
>>> 
>>> Cheers,
>>> 
>>> Derek
>>> 
>>> On Thu, Mar 9, 2023 at 6:44 AM Miklosovic, Stefan 
>>> mailto:stefan.mikloso...@netapp.com>> 
>>> wrote:
>>> Hi list,
>>> 
>>> I stumbled upon Hadoop package again. I think there was some discussion 
>>> about the relevancy of Hadoop code some time ago but I would like to 
>>> ask this again.
>>> 
>>> Do you think Hadoop code (1) is still relevant in 5.0? Who in the 
>>> industry is still using that?
>>> 
>>> We might drop a lot of code and some Hadoop dependencies too (3) (even 
>>> their scope is "provided"). The version of Hadoop we build upon is 
>>> 1.0.3 which was released 10 years ago. This code does not have any 
>>> tests nor documentation on the website.
>>> 
>>> There seems to be issues like this (2) and it seems like the solution 
>>> is to, basically, use Spark Cassandra connector instead which I would 
>>> say is quite reasonable.
>>> 
>>> Regards
>>> 
>>> (1) 
>>> https://github.com/apache/cassandra/tree/trunk/src/java/org/apache/cassandra/hadoop
>>> (2) https://lists.apache.org/thread/jdy5hdc2l7l29h04dqol5ylroqos1y2p
>>> (3) 
>>> https://github.com/apache/cassandra/blob/trunk/.build/parent-pom-template.xml#L507-L589
>>> 
>>> 
>>> --
>>> +---+
>>> | Derek Chen-Becker |
>>> | GPG Key available at 
>>> 

Re: [VOTE] CEP-21 Transactional Cluster Metadata

[Jeremy Hanna]

I just want to say after Alex's presentation at ApacheCon, I'm really excited 
about this as an underappreciated feature for 5.0: how it will help cluster 
stability through membership changes and schema evolutions, how it will improve 
both the scalability of clusters and the flexibility of replica placement, and 
some interesting possibilities once this foundation is in place.  For any 
interested, Alex's talk was recorded and is found here: 
https://drive.google.com/file/d/1LKNpp3rnBlHnls3HUMRZAL9W1llylN0a/view?usp=sharing

> On Feb 10, 2023, at 2:46 AM, Sam Tunnicliffe  wrote:
> 
> The vote passes with 23 +1s (13 binding) and no negative votes.
> 
> Thanks all,
> Sam
> 
> 
>> On 6 Feb 2023, at 16:15, Sam Tunnicliffe  wrote:
>> 
>> Hi everyone,
>> 
>> I would like to start a vote on this CEP.
>> 
>> Proposal:
>> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-21%3A+Transactional+Cluster+Metadata
>> 
>> Discussion:
>> https://lists.apache.org/thread/h25skwkbdztz9hj2pxtgh39rnjfzckk7
>> 
>> The vote will be open for 72 hours.
>> A vote passes if there are at least three binding +1s and no binding vetoes.
>> 
>> Thanks,
>> Sam
> 

Re: [DISCUSS] CEP-22: Drivers Donation Status

[Jeremy Hanna]

At DataStax, we're trying to make sure we have everything in order from a CLA 
perspective before moving forward with the process.  The drivers have always 
been Apache-2.0 licensed but we just want to make sure everything is in order.  
We're really close on the Java driver to move forward.  Once that's done, my 
understanding is that it will go through the next phases of the process and I 
know Josh and others are looking further into how we want to organize 
committers for the drivers.  We're doing the work for all of the drivers but 
will start with Java and then Python because of project dependencies and to 
learn from that process for the others.

> On Jan 12, 2023, at 12:36 PM, Abe Ratnofsky  wrote:
> 
> What's the current status of CEP-22?
> 
> It looks like the CEP draft on Confluence [1] hasn't been updated since Aug 
> 2022 and mailing list search [2] shows no results, so it never made it to a 
> vote.
> 
> --
> Abe
> 
> [1]: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-22%3A+Datastax+Drivers+Donation
> [2]: https://lists.apache.org/list?*@cassandra.apache.org:lte=240M:CEP-22
> 
> 

Re: Naming conventions for CQL native functions

[Jeremy Hanna]

+1 (nb) mixed case is a miserable experience and snake case makes it readable. 

> On Nov 10, 2022, at 10:57 AM, Francisco Guerrero  wrote:
> 
> +1 (nb) as well
> 
>> On 2022/11/10 17:16:21 Caleb Rackliffe wrote:
>> +100 on snake case for built-in functions  given I think MySQL and Postgres
>> use that convention as well.
>> 
>> ex. https://www.postgresql.org/docs/9.2/functions-string.html
>> 
>>> On Thu, Nov 10, 2022 at 7:51 AM Brandon Williams  wrote:
>>> 
>>> I too meant snake case and need coffee.
>>> 
 On Thu, Nov 10, 2022, 7:26 AM Brandon Williams  wrote:
>>> 
 +1 on camel case and aliases for compatibility.
 
 On Thu, Nov 10, 2022, 6:21 AM Andrés de la Peña 
 wrote:
 
> It seems we don't have a clear convention on how to name CQL native
> functions.
> 
> Most native functions are named all lower case, without underscore nor
> hyphen to separate words. That's the case, for example, of "intasblob" or
> "blobasint".
> 
> We also have some functions using camel case, as in "castAsInt" or
> "castAsTimestamp". Note that the came cased names require quoting due to
> CQL's case insensitivity.
> 
> Differently to CQL native functions, system keyspaces, tables and
> columns consistently use snake case. For example, we have "system_schema",
> "dropped_columns", "default_time_to_live".
> 
> I think it would be good to adopt a convention on how to name CQL native
> functions, at least the new ones. IMO camel case would make sense because
> it plays well with CQL's case insensitivity, it makes long names easier to
> read and it's consistent with the names used for most other things.
> 
> For example, in CASSANDRA-17811 I'm working on a set of functions to do
> within-collection operations, which would be named "map_keys",
> "map_values", "collection_min", "collection_max", "collection_sum",
> "collection_count", etc. Also, CEP-20 will add a set of functions that
> would be named "mask_null", "mask_default", "mask_replace", "mask_inner",
> "mask_outer", "mask_hash", etc.
> 
> As for the already existing functions, we could either let them be or
> add snake case aliases for them, so for example we'd have both "castAsInt"
> and "cast_as_int", at least for a time.
> 
> What do you think?
> 
 
>> 

Re: Thanks to Nate for his service as PMC Chair

[Jeremy Hanna]

Thanks Nate for all of your behind the scenes work!  Thanks Mick for stepping 
up even more to help with the project!

> On Jul 11, 2022, at 7:54 AM, Paulo Motta  wrote:
> 
> Hi,
> 
> I wanted to announce on behalf of the Apache Cassandra Project Management 
> Committee (PMC) that Nate McCall (zznate) has stepped down from the PMC chair 
> role. Thank you Nate for all the work you did as the PMC chair!
> 
> The Apache Cassandra PMC has nominated Mick Semb Wever (mck) as the new PMC 
> chair. Congratulations and good luck on the new role Mick!
> 
> The chair is an administrative position that interfaces with the Apache 
> Software Foundation Board, by submitting regular reports about project status 
> and health. Read more about the PMC chair role on Apache projects:
> - https://www.apache.org/foundation/how-it-works.html#pmc 
> 
> - https://www.apache.org/foundation/how-it-works.html#pmc-chair 
> 
> - https://www.apache.org/foundation/faq.html#why-are-PMC-chairs-officers 
> 
> 
> The PMC as a whole is the entity that oversees and leads the project and any 
> PMC member can be approached as a representative of the committee. A list of 
> Apache Cassandra PMC members can be found on: 
> https://cassandra.apache.org/_/community.html 
> 
> 
> Kind regards,
> 
> Paulo

Re: New Apache Cassandra Group on LinkedIn

[Jeremy Hanna]

Is it possible to ask someone at linkedin to merge the groups together so that 
it's managed by the PMC but with the explicit permission of the people running 
the other group?  In the past, I know that Twitter does things like that in 
terms of handles and followers.  Is that a desirable outcome?

> On Mar 9, 2022, at 11:00 AM, Benjamin Lerer  wrote:
> 
> Hi Patrick,
> Thanks for reaching out. Effectively the discussion has happened between the 
> PMC members.
> To explain the context, we wanted to have an official group on Linkedin to 
> publish news about the project as we do through the @cassandra handler on 
> Twitter. We wanted a group that was vendor independent and focused on Apache 
> Cassandra and its ecosystem.
> To be fully transparent, we had no idea that you were in charge of the Apache 
> Cassandra Users group as it appears managed by Lynn Bender and Joanna Kapel. 
> The group also appears to promote different vendors which is something that 
> we wanted to avoid.
> Having to post things under Lynn's name was also an issue for us as we wished 
> the merits to go to the right persons.
> 
> Now, I am sure that we can work out some solution that will benefit the 
> community. :-)
> 
> Le mer. 9 mars 2022 à 15:56, Patrick McFadin  > a écrit :
> I feel like this needs to be a discussion held on the public mailing list. I 
> have been running the Apache Cassandra Users group on LinkedIn for years 
> after taking it over from Lynn Bender. 
> https://www.linkedin.com/groups/3803052/ 
> 
> 
> We have over 7500 members and had its ups and downs but it's been pretty 
> consistent as a professional resource on LinkedIn. I'm not sure what there is 
> to gain by creating competing groups. If we need more managers in the group 
> that's fine but somebody just needed to ask. It's clear that this discussion 
> happened somewhere else and this was just an announcement. 
> 
> Patrick
> 
> On Thu, Mar 3, 2022 at 3:41 AM Benjamin Lerer  > wrote:
> Hi everybody,
> 
> We just created a new Apache Cassandra group on LinkedIn 
> (https://www.linkedin.com/groups/9159443/ 
> ).
> 
> This group will be managed by our community and will respect vendor 
> neutrality.
> Do not hesitate to join and share your experiences or blog posts with us :-)

Re: [DISCUSS] CASSANDRA-17292 Move cassandra.yaml toward a nested structure around major database concepts

[Jeremy Hanna]

If we support both formats for a time, I just would want to make absolutely 
sure that it will read only one or the other so there's no uncertainty about 
the server configuration.  Perhaps to avoid unforeseen migration problems, we 
only read the old format if a specific flag is set?  So with version 5, we only 
read the new format by default.  So if you only have the old format and you try 
to start 5.0, it will fail with a log message about a JVM option to be used 
("READ_CASSANDRA_YAML" or something).  So if you enable that, you *only* read 
the old config.  It would be one or the other so you don't have weird dilemmas 
of which one to choose.

> On Feb 23, 2022, at 11:30 AM, Caleb Rackliffe  
> wrote:
> 
> 
> Continuing to parse the old format for some time seems unavoidable, and 
> allowing dot-separated options in the old format seems reasonable.
> 
> There will certainly be some interesting problems when we move into 
> implementation space with this. One approach might be to implement a clean 
> object model that corresponds to the new format, work out how it's 
> parsed/populated from the file, and then have some kind of converter from the 
> old Config object to the new object model that allows us to provide values to 
> DatabaseDescriptor from only the new one (thereby avoiding any changes to the 
> places all over the codebase that use DD).
> 
> On Wed, Feb 23, 2022 at 4:46 AM Bowen Song  > wrote:
> I agree with Benedict, there's legit use cases for both the flat and 
> structured config file format. The operator should be able to choose which 
> one is best suited for their own use case. It will also make the upgrade 
> process easier if both formats are supported by future versions of Cassandra.
> 
> On 23/02/2022 07:52, bened...@apache.org  wrote:
>> I agree that a new configuration layout should be introduced once only, not 
>> incrementally.
>> 
>>  
>> 
>> However, I disagree that we should immediately deprecate the old config file 
>> and refuse to parse it. We can maintain compatibility indefinitely at low 
>> cost, so we should do so.
>> 
>>  
>> 
>> Users of the old format, when using new configuration options, can simply 
>> use dot separators to specify them. Since most settings are not required, 
>> this is by far the least painful upgrade process.
>> 
>>  
>> 
>>  
>> 
>> From: Berenguer Blasi  
>> 
>> Date: Wednesday, 23 February 2022 at 06:53
>> To: dev@cassandra.apache.org  
>>  
>> Subject: Re: [DISCUSS] CASSANDRA-17292 Move cassandra.yaml toward a nested 
>> structure around major database concepts
>> 
>> +1 to a non-incremental approach as well.
>> 
>> On 23/2/22 1:27, Caleb Rackliffe wrote:
>> > @Patrick I’m absolutely intending for this to be a 5.0 concern. The only 
>> > reason why it would have any bearing on 4.x is the case where we’re adding 
>> > new config that could fit into the v2 structure now and not require any 
>> > later changes.
>> >
>> >> On Feb 22, 2022, at 3:22 PM, Bernardo Sanchez 
>> >>   
>> >> wrote:
>> >>
>> >> unsubscribe
>> >>
>> >> -Original Message-
>> >> From: Stefan Miklosovic  
>> >> 
>> >> Sent: Tuesday, February 22, 2022 3:53 PM
>> >> To: dev@cassandra.apache.org 
>> >> Subject: Re: [DISCUSS] CASSANDRA-17292 Move cassandra.yaml toward a 
>> >> nested structure around major database concepts
>> >>
>> >> "EXTERNAL EMAIL" - This email originated from outside of the 
>> >> organization. Do not click or open attachments unless you recognize the 
>> >> sender and know the content is safe. If you are unsure, please contact 
>> >> hel...@pointclickcare.com .
>> >>
>> >> I want to add that to, however, on the other hand, we also do have dtests 
>> >> in Python and they need to run with old configs too. That is what 
>> >> Ekaterina was doing - supporting old configuration while introducing new 
>> >> one. If we make "a big cut" and old way of doing things would not be 
>> >> possible, how are we going to treat this in dtests when we will have 
>> >> stuff for 3.11, 4 on old configs and 5 on new configs?
>> >>
>> >>> On Tue, 22 Feb 2022 at 21:48, Stefan Miklosovic 
>> >>>  
>> >>>  wrote:
>> >>>
>> >>> +1 to what Patrick says.
>> >>>
>>  On Tue, 22 Feb 2022 at 21:40, Patrick McFadin  
>>   wrote:
>> 
>>  I'm going to put up a red flag of making config file changes of this 
>>  scale on a dot release. This should really be a 5.0 consideration.
>> 
>>  With that, I would propose a #5. 5.0 nodes will only read the new 
>>  config files and reject old config files. If any of you went through 
>>  the config file changes from Apache HTTPd 1.3 -> 2.0 you 

Re: [VOTE] CEP-7: Storage Attached Index

[Jeremy Hanna]

+1 nb. Thanks Caleb, Mike, Jason, and everyone involved with the effort.

> On Feb 17, 2022, at 4:23 PM, Caleb Rackliffe  wrote:
> 
> 
> Hi everyone,
> 
> I'd like to call a vote to approve CEP-7.
> 
> Proposal: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-7%3A+Storage+Attached+Index
> 
> Discussion: https://lists.apache.org/thread/hh67k3t86m7299qkt61gmzb4h96bl90w
> 
> The vote will be open for 72 hours.
> Votes by committers are considered binding.
> A vote passes if there are at least three binding +1s and no binding vetoes.
> 
> Thanks!
> Caleb

Re: [VOTE] CEP-19: Trie memtable implementation

[Jeremy Hanna]

+1 nb.  Thanks for all of the great work on this Branimir.  Excited to see this 
moving forward.

> On Feb 16, 2022, at 7:56 AM, J. D. Jordan  wrote:
> 
> +1 nb
> 
>> On Feb 16, 2022, at 7:30 AM, Josh McKenzie  wrote:
>> 
>> 
>> +1
>> 
>> On Wed, Feb 16, 2022, at 7:33 AM, Ekaterina Dimitrova wrote:
>>> +1nb
>>> 
>>> On Wed, 16 Feb 2022 at 7:30, Brandon Williams >> > wrote:
>>> +1
>>> 
>>> On Wed, Feb 16, 2022 at 3:00 AM Branimir Lambov >> > wrote:
>>> >
>>> > Hi everyone,
>>> >
>>> > I'd like to propose CEP-19 for approval.
>>> >
>>> > Proposal: 
>>> > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-19%3A+Trie+memtable+implementation
>>> >  
>>> > 
>>> > Discussion: 
>>> > https://lists.apache.org/thread/fdvf1wmxwnv5jod59jznbnql23nqosty 
>>> > 
>>> >
>>> > The vote will be open for 72 hours.
>>> > Votes by committers are considered binding.
>>> > A vote passes if there are at least three binding +1s and no binding 
>>> > vetoes.
>>> >
>>> > Thank you,
>>> > Branimir

Re: [DISCUSS] Java support roadmap

[Jeremy Hanna]

One of the things I'm excited about with the 4+ releases is this running with 
Java 17+ (including ZGC) so I would love to see this get more serious testing 
and validation.

Anecdotal account - at an event a couple of years ago, I spoke to someone from 
an education software company was running some of their clusters on Java 11 (or 
possibly a later version) and it ran really well for them.

> On Aug 27, 2021, at 12:35 AM, Ekaterina Dimitrova  
> wrote:
> 
> Hi everyone,
> I had a few people asking me recently about the Java 11 support.
> I came across this thread we had last year -
> https://lists.apache.org/thread.html/r38f6beaa22e247cb38212f476f5f79efdc6587f83af0397406c06d7c%40%3Cdev.cassandra.apache.org%3E
> .
> I think we have all tests running in both Java 8 and Java 11 for trunk and
> cassandra-4.0 in both Jenkins and CircleCI.
> Does anyone know about any Java 11 specific bugs discovered? I personally
> haven't heard of any in the past year. What is the plan of the community
> for moving Java 11 support out of experimental?
> Also, Java 17 LTS GA is planned for Sepember 2021 and I can try to test it
> with trunk.
> Any thoughts?
> Last but not least, do we know  anyone running Java 11 in production?
> This thread was really opened as a stage to share our thoughts and
> hopefully come up with a plan as a community.
> 
> Looking forward to seeing what people think here.
> 
> Thank you,
> Ekaterina


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSS] CEP 14: Paxos Improvements

[Jeremy Hanna]

It sounds like a great improvement!

Just for those who had followed the development of ePaxos* that Blake and 
others worked on but was never committed, it would be nice to briefly compare 
the two. 

https://issues.apache.org/jira/browse/CASSANDRA-6246

> On Aug 19, 2021, at 9:18 AM, Scott Andreas  wrote:
> 
> Benedict, thank you for sharing this CEP!
> 
> Adding some notes on why I support this proposal:
> 
> - Reducing common-case round trips from 4x to 2x on writes and 2x to 1x on 
> reads is a huge improvement. This latency reduction may be sufficient to 
> allow many users of Cassandra who operate in a single datacenter, 
> availability zone, or region to migrate to a multi-region topology.
> 
> - The Cluster Simulation work described in CEP-10 provides a toolchain for 
> probabilistically-exhaustive validation and simulation of transactional 
> correctness, allowing assertion of linearizability in the presence of 
> adversarial thread scheduling and message ordering over an unbounded number 
> of simulated clusters and transactions.
> 
> - Some use cases may see a superlinear increase in LWT performance due to a 
> reduction in contention afforded by fewer message round-trips. E.g., halving 
> latency shortens the interval during which competing transactions may 
> conflict, reducing contention and improving throughput beyond a level that 
> would be afforded by the latency reduction alone.
> 
> - Better safety among range movements: Electorate verification during range 
> movements provides a stronger assertion of linearizability via assurance of 
> the set of instances voting on a transaction.
> 
> – Scott
> 
> 
> From: bened...@apache.org 
> Sent: Wednesday, August 18, 2021 2:31 PM
> To: dev@cassandra.apache.org
> Subject: [DISCUSS] CEP 14: Paxos Improvements
> 
> RE: 
> https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-14%3A+Paxos+Improvements
> 
> I’m proposing this CEP for approval by the project. The goal is to both 
> improve the performance of LWTs and to ensure their correctness across a 
> range of scenario like range movements. This work builds upon the Simulator 
> CEP that has been recently adopted, and patches will follow in the coming 
> weeks.
> 
> If you have any concerns or questions please raise them here for discussion.
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

Re: [VOTE] Release Apache Cassandra 4.0.0 (third time is the charm)

[Jeremy Hanna]

+1 nb

On Sat, Jul 24, 2021 at 11:50 AM Scott Andreas  wrote:

> +1 nb
>
> > On Jul 23, 2021, at 2:46 PM, Mick Semb Wever  wrote:
> >
> > 
> >>
> >>
> >> The vote will be open for 72 hours (longer if needed). Everyone who
> >> has tested the build is invited to vote. Votes by PMC members are
> >> considered binding. A vote passes if there are at least three binding
> >> +1s and no -1's.
> >>
> >>
> >
> > +1
> >
> >
> > Tested:
> > sha1, md5, sha256, sha512 match.
> > gpg signatures match.
> > Source artefact builds. (and doesn't contain compiled dependencies)
> > Tarball Binary artefact runs.
> > Debian binary package runs.
> > Redhat binary package runs.
>

Re: [VOTE] Release Apache Cassandra 4.0.0 (take2)

[Jeremy Hanna]

+1 (nb)

> On Jul 15, 2021, at 3:42 AM, Blake Eggleston  
> wrote:
> 
> +1
> 
>> On Jul 14, 2021, at 8:21 AM, Aleksey Yeschenko  wrote:
>> 
>> +1
>> 
 On 14 Jul 2021, at 15:37, Jonathan Ellis  wrote:
>>> 
>>> +1
>>> 
 On Tue, Jul 13, 2021 at 5:14 PM Mick Semb Wever  wrote:
 
 Proposing the test build of Cassandra 4.0.0 for release.
 
 sha1: 924bf92fab1820942137138c779004acaf834187
 Git:
 https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0.0-tentative
 Maven Artifacts:
 
 https://repository.apache.org/content/repositories/orgapachecassandra-1242/org/apache/cassandra/cassandra-all/4.0.0/
 
 The Source and Build Artifacts, and the Debian and RPM packages and
 repositories, are available here:
 https://dist.apache.org/repos/dist/dev/cassandra/4.0.0/
 
 The vote will be open for 72 hours (longer if needed). Everyone who
 has tested the build is invited to vote. Votes by PMC members are
 considered binding. A vote passes if there are at least three binding
 +1s and no -1's.
 
 [1]: CHANGES.txt:
 
 https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0.0-tentative
 [2]: NEWS.txt:
 https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0.0-tentative
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
 
>>> 
>>> -- 
>>> Jonathan Ellis
>>> co-founder, http://www.datastax.com
>>> @spyced
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-rc2

[Jeremy Hanna]

+1 (nb) nice to see all of the fixes and the use of newer TLS by default and 
obfuscation of passwords in the audit log for the 4.0 release.

> On Jun 29, 2021, at 6:01 AM, Jon Meredith  wrote:
> 
> +1 (nb)
> 
>> On Mon, Jun 28, 2021 at 9:47 AM Yifan Cai  wrote:
>> 
>> +1
>> 
>> 
>> - Yifan
>> 
 On Jun 28, 2021, at 8:40 AM, Ekaterina Dimitrova  
 wrote:
>>> 
>>> +1 Thanks everyone!
>>> 
 On Mon, 28 Jun 2021 at 11:39, Aleksey Yeschenko  wrote:
 
 +1
 
>> On 28 Jun 2021, at 14:05, Gary Dusbabek  wrote:
> 
> +1; yay!
> 
>> On Sun, Jun 27, 2021 at 11:02 AM Mick Semb Wever  wrote:
> 
>> Proposing the test build of Cassandra 4.0-rc2 for release.
>> 
>> sha1: 4c98576533e1d7663baf447e8877788096489165
>> Git:
>> 
>> 
 https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-rc2-tentative
>> Maven Artifacts:
>> 
>> 
 https://repository.apache.org/content/repositories/orgapachecassandra-1237/org/apache/cassandra/cassandra-all/4.0-rc2/
>> 
>> The Source and Build Artifacts, and the Debian and RPM packages and
>> repositories, are available here:
>> https://dist.apache.org/repos/dist/dev/cassandra/4.0-rc2/
>> 
>> The vote will be open for 72 hours (longer if needed). Everyone who has
>> tested the build is invited to vote. Votes by PMC members are considered
>> binding. A vote passes if there are at least three binding +1s and no
 -1's.
>> 
>> [1]: CHANGES.txt:
>> 
>> 
 https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-rc2-tentative
>> [2]: NEWS.txt:
>> 
>> 
 https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=NEWS.txt;hb=refs/tags/4.0-rc2-tentative
>> [3]: The maven artifacts were accidentally prematurely made public. Docs
>> have been updated to prevent this happening again.
>> 
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Welcome Caleb Rackliffe as Cassandra committer

[Jeremy Hanna]

Congratulations Caleb!

> On May 14, 2021, at 11:02 PM, Mick Semb Wever  wrote:
> 
> The PMC members are pleased to announce that Caleb Rackliffe has
> accepted the invitation to become committer.
> 
> Thanks heaps Caleb for helping make Cassandra awesome!
> 
> Congratulations and welcome,
> The Apache Cassandra PMC members


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSSION] Attracting new contributors

[Jeremy Hanna]

I believe Paolo started with the project through a contributor boot camp. Also 
if I remember correctly some of the ones that were done were internal at 
DataStax and it helped some people get familiar with the project who still 
contribute today. 

Also this would be short recorded introductions so they could be around for 
viewing and with auto translate on Google for different languages such as 
Japanese and Mandarin. 

I do like the idea of a periodic chat. I just thought some recorded 
introductions would help with some of the more common things like “this is how 
the read path works from end to end”. 

> On Apr 27, 2021, at 10:14 PM, Benedict Elliott Smith  
> wrote:
> 
> I think that all of the bootcamps we ran in the past produced precisely zero 
> new contributors.
> 
> I wonder if it would be more impactful to produce slightly more permanent 
> content, such as step-by-step guides to producing a simple patch for some 
> subsystem. Perhaps if people want to, a recording could be created of going 
> through that guide as well.
> 
> That said, if there are new contributors actively trying to participate, 
> organising a periodic group chat to talk through one of the issues that they 
> may be working on together as a group with an active contributor might make 
> sense, and be more targeted in focus?
> 
> 
> On 27/04/2021, 12:45, "Manish G"  wrote:
> 
>Contributor bootcamps can really help new people like me.
> 
>>On Tue, Apr 27, 2021, 5:08 PM Jeremy Hanna 
>>wrote:
>> 
>> One thing we've done in the past is contributor bootcamps along with the
>> the new contributor guide and the LHF complexity tickets.  Unfortunately, I
>> don't know that the contributor bootcamps were ever recorded.
>> Presentations were done to introduce people to the codebase generally (I
>> think Gary did this at one point) as well as specific parts of the
>> codebase, such as compaction.  What if we broke up the codebase into
>> categories and people could volunteer to do a short introduction to that
>> part of the codebase in the form of a video screenshare.  I don't think
>> this would take the place of mentoring someone, but if we had introductions
>> to different parts of the codebase, I think it would lower the bar for
>> interested contributors and scale the existing group more easily.  Besides
>> the codebase itself, we could also introduce things like CI practices or
>> testing or documentation.
>> 
>>>> On Apr 24, 2021, at 12:49 AM, Benjamin Lerer  wrote:
>>> 
>>> Hi Everybody,The Apache Cassandra project always had some issues to
>>> attract and retain new contributors. I think it would be great to change
>>> this.According to the "How to Attract New Contributors" blog post (
>>> https://www.redhat.com/en/blog/how-attract-new-contributors) having a
>> good
>>> onboarding process is a critical part. How to contribute should be
>> obvious
>>> and contributing should be as easy as possible for all the different
>> types
>>> of contributions: code, documentation, web-site or help with our CI
>>> infrastructure.I would love to hear about your ideas on how we can
>> improve
>>> things.If you are new in the community, do not hesitate to share your
>>> experience and your suggestions on what we can do to make it easier for
>> you
>>> to contribute.
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [DISCUSSION] Attracting new contributors

[Jeremy Hanna]

One thing we've done in the past is contributor bootcamps along with the the 
new contributor guide and the LHF complexity tickets.  Unfortunately, I don't 
know that the contributor bootcamps were ever recorded.  Presentations were 
done to introduce people to the codebase generally (I think Gary did this at 
one point) as well as specific parts of the codebase, such as compaction.  What 
if we broke up the codebase into categories and people could volunteer to do a 
short introduction to that part of the codebase in the form of a video 
screenshare.  I don't think this would take the place of mentoring someone, but 
if we had introductions to different parts of the codebase, I think it would 
lower the bar for interested contributors and scale the existing group more 
easily.  Besides the codebase itself, we could also introduce things like CI 
practices or testing or documentation.

> On Apr 24, 2021, at 12:49 AM, Benjamin Lerer  wrote:
> 
> Hi Everybody,The Apache Cassandra project always had some issues to
> attract and retain new contributors. I think it would be great to change
> this.According to the "How to Attract New Contributors" blog post (
> https://www.redhat.com/en/blog/how-attract-new-contributors) having a good
> onboarding process is a critical part. How to contribute should be obvious
> and contributing should be as easy as possible for all the different types
> of contributions: code, documentation, web-site or help with our CI
> infrastructure.I would love to hear about your ideas on how we can improve
> things.If you are new in the community, do not hesitate to share your
> experience and your suggestions on what we can do to make it easier for you
> to contribute.


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Supported upgrade path for 4.0

[Jeremy Hanna]

So to reiterate the recommended/tested paths that I get from this thread:

2.1.x -> 2.1.latest -> [3.0.latest | 3.11.latest] -> 4.0
2.2.x -> 2.2.latest -> [3.0.latest | 3.11.latest] -> 4.0
3.0 -> 3.0.latest -> 4.0
3.x -> 3.11.latest -> 4.0

I just wanted to be explicit about getting to the latest in the current line 
you're in before upgrading to reduce risks and test surface area by upgrading 
from a single '.latest' version.  That and in case there are additional fixes 
added in this upgrade test process to make the '.latest' version more stable 
for the upgrade.  Also we hadn't mentioned 2.2 which some people are running 
and is currently supported by the project.

Something that makes [3.0.latest | 3.11.latest] as a midpoint less of a risk is 
that both 3.0.latest and 3.11.latest use the same sstable version (md).  That 
sstable version came about in 3.0.18/3.11.4.  If people are on earlier versions 
of 3.0 or 3.x, we should consider recommending that they upgrade to 3.0.latest 
or 3.11.latest, run upgradessttables, and then go to 4.0 to just make sure they 
go from md to na (4.0).  Just to save people from looking it up, the last major 
change to the sstable format was in 3.0.18/3.11.4 with version md to correct 
sstable min/max metadata.  See 
https://issues.apache.org/jira/browse/CASSANDRA-14861 and 
https://github.com/apache/cassandra/blob/cassandra-3.11/src/java/org/apache/cassandra/io/sstable/format/big/BigFormat.java#L128.
  I've been surprised to see some clusters that back API gateways running 3.7 
for example.

> On Oct 12, 2020, at 8:32 AM, Scott Andreas  wrote:
> 
> Great, thanks Ben!
> 
> The primary configuration my colleagues and I will be vetting is the 3.0.x -> 
> 4.0 path (implicitly, 2.1 -> 3.0 -> 4.0). From a quality + safety perspective 
> we will be ensuring that it’s a smooth ride for folks who opt for this route; 
> though no major concerns on my part with the project recommending 2.1 -> 3.11 
> -> 4.0 aside from not having experienced it myself.
> 
>> On Oct 11, 2020, at 12:47 AM, Ben Slater  wrote:
>> 
>> Just to add to Mick's point, we (Instaclustr) have also been running and
>> recommending 3.11.x by default. It's currently by far the most common
>> version in our managed fleet and our last 3.0.x cluster will likely be
>> upgraded shortly. 3.11.x is also our recommendation for consulting and
>> support customers. I'd therefore support Mick's recommendation (really
>> based on our experience with and confidence in 3.11.x rather than being
>> able to point to specific issues off hand) that 2.*->3.11.x->4.0 is the
>> preferred upgrade path. We will do testing on 3.11.x to 4.0 upgrade but I
>> can't see us doing any work on 3.0 to 4.0.
>> 
>> Cheers
>> Ben
>> 
>> ---
>> 
>> 
>> *Ben Slater**Chief Product Officer*
>> 
>> 
>> 
>>    
>> 
>> 
>> Read our latest technical blog posts here
>> .
>> 
>> This email has been sent on behalf of Instaclustr Pty. Limited (Australia)
>> and Instaclustr Inc (USA).
>> 
>> This email and any attachments may contain confidential and legally
>> privileged information.  If you are not the intended recipient, do not copy
>> or disclose its content, but please reply to this email immediately and
>> highlight the error to the sender and then immediately delete the message.
>> 
>> 
>> On Sun, 11 Oct 2020 at 06:42, Mick Semb Wever  wrote:
>> 
 "3.11 performs close to parity with 2.1/2.2. 3.0 does not. If we
>>> recommend
 people upgrade from 2.1 -> 3.0 -> 4.0, we are asking them to have a
>>> cluster
 in a regressed performance state for potentially months as they execute
 their upgrade."
 
 Did I get anything wrong here Mick? ^
 
>>> 
>>> 
>>> That's correct Josh.
>>> 
>>> From tickets like those listed, and from experience, we recommend folk
>>> avoid 3.0 altogether. This has only been made more evident by witnessing
>>> the benefits from 3.0 → 3.11 upgrades.
>>> 
>>> My recommendation remains  2.*→3.11→4.0. And I don't believe I'm alone.
>>> Though if a user was already on 3.0, then I would (of course) recommend an
>>> upgrade directly to 4.0.
>>> 
>>> I feel like I'm just splitting straws at this point, since we have accepted
>>> (folk willing to help with) both paths to 4.0, and I can't see how we stop
>>> recommending  2.*→3.11 upgrades.
>>> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Committing `CASSANDRA-13701 Lower default num_tokens` and the dtest slowdown…

[Jeremy Hanna]

Thanks Adam.

Now that we have CASSANDRA-16079 created and work is proceeding, can we please 
commit CASSANDRA-13701?  There are many clusters out there that use the default 
of 256 num_tokens and migrating to something more sane is a lot of work.  It 
would also be very helpful for having reasonable defaults getting tested for 
the release.  It's been a bad default for a long, long time and we need to get 
this in before we do more testing for the release.

> On Aug 28, 2020, at 5:25 AM, Adam Holmberg  wrote:
> 
> After discussing with a few stakeholders it sounds like folks agree that
> optimizing dtest speed is a worthy endeavor. What is less clear are
> concrete things that should be done. Since a brainstorming session failed
> to materialize on CASSANDRA-13701, we thought it would make sense to start
> with an open analysis.
> 
> A ticket[1] has been created for analysis and further follow-up. We welcome
> any concrete ideas people already have in mind.
> 
> Kind regards,
> Adam Holmberg
> 
> [1] https://issues.apache.org/jira/browse/CASSANDRA-16079
> 
> On Mon, Aug 24, 2020 at 12:17 PM Mick Semb Wever  wrote:
> 
>> I believe the speed of our CI runs
>>> is of common interest. What do you think? Does this sound feasible? Who
>> is
>>> in?*
>>> 
>> 
>> 
>> I agree. I'm in. I have some ideas to add to the mix. Thanks Ekaterina.
>> 
> 
> 
> -- 
> Adam Holmberg
> e. adam.holmb...@datastax.com
> w. www.datastax.com


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Committing `CASSANDRA-13701 Lower default num_tokens` and the dtest slowdown…

[Jeremy Hanna]

I know the dtests take a long time and this will make them longer. As a counter 
point most people run individual dtests locally and the full set on dedicated 
test infrastructure. For the dedicated test infrastructure Mick also improved 
the wall clock runtime when parallelizing the dtests on 
https://issues.apache.org/jira/browse/CASSANDRA-16006. 

Even with the longer dtest full runtime, I firmly believe that for the sake of 
new users and how hard it is to change num_tokens once data is written, this 
change to the default of num_tokens is long overdue. Another hidden benefit of 
this change is that the dtests will now run bootstraps the way operators should 
run them in practice with the new defaults. That will make the more common 
default case much more tested and hopefully catch regressions in that execution 
path faster.

So while it is not a trivial change in full dtest runtime, the benefits to the 
community and project are also not trivial. I’m really grateful to all who have 
put in effort to make this a reality and know that new users in 4.0 will 
benefit from these improved defaults.

In other words my non binding vote is to merge this and look to improve 
execution time separately with that effort not being as urgent for the reasons 
stated above.

Jeremy

> On Aug 20, 2020, at 2:49 AM, Mick Semb Wever  wrote:
> 
> It was agreed¹ that 4.0 should have the new configuration defaults of
>  num_tokens: 16
>  allocate_tokens_for_local_replication_factor: 3
> 
> 13701's patches: against cassandra, cassandra-builds, cassandra-dtest, ccm;
> are reviewed, tested, and ready to commit. But the ccm and dtest patches
> required ccm having to now start nodes sequentially, and adding some longer
> timeout values in the dtests.
> 
> The consequence of this is CI runs now take longer. ci-cassandra.a.o's
> dtests take ~30% longer, and circleci's dtests (with vnodes) have gone from
> ~22 to ~43 minutes. The general opinion (on slack²) is to commit, and work
> on improving ccm and dtest startup times in a subsequent ticket.
> 
> 13701 was intended to be committed before the first beta release because of
> its user-facing changes. But these numbers are significant enough it makes
> sense to touch base with dev@
> 
> Does anyone (strongly) object to the "commit + follow up ticket" approach?
> 
> regards,
> Mick
> 
> 
> ¹ –
> https://lists.apache.org/thread.html/ra829084fcf344e9e96fa5c61cb31e909c8629091993471594b65ea89%40%3Cdev.cassandra.apache.org%3E
> ² – https://the-asf.slack.com/archives/CK23JSY2K/p1597747395032600 and
> https://the-asf.slack.com/archives/CK23JSY2K/p1597849774078200?thread_ts=1597762085.048300=CK23JSY2K

Re: [Discuss] num_tokens default in Cassandra 4.0

[Jeremy Hanna]

Just to close the loop on this,
https://issues.apache.org/jira/browse/CASSANDRA-13701 is getting tested
now.  The project testing will get updated to utilize the new defaults
(both num_tokens and using the new allocation algorithm by uncommenting
allocate_tokens_for_local_replication_factor: 3.  Jon did some
documentation on num_tokens on
https://cassandra.apache.org/doc/latest/getting_started/production.html#tokens
on a separate ticket he mentioned -
https://issues.apache.org/jira/browse/CASSANDRA-15600.  The new default in
Cassandra 4.0+ will be to use the new allocation algorithm with num_tokens:
16.  There is a note in the NEWS.txt about upgrading and bootstrapping.  It
is a lot of effort to change this once it is set, so hopefully new users
will be in a much better place out of the box.  Thanks everyone for your
efforts in this.

On Wed, Apr 1, 2020 at 4:28 PM Jeremy Hanna 
wrote:

> As discussed, let's go with 16.  Speaking with Anthony privately as well,
> I had forgotten that some of the analysis that Branimir had initially done
> on the skew and allocation may have been internal to DataStax so I should
> have mentioned that previously.  Thanks to Mick, Alex, and Anthony for
> doing this analysis and helping back the decision with data.  This will
> benefit many that start with Cassandra that don't know that 256 is a bad
> number and end up with a hard to change decision later.  I assigned myself
> to https://issues.apache.org/jira/browse/CASSANDRA-13701.  Thanks all.
>
> On Wed, Mar 11, 2020 at 6:02 AM Mick Semb Wever  wrote:
>
>>
>>
>> > I propose we drop it to 16 immediately.  I'll add the production docs
>> > in CASSANDRA-15618 with notes on token count, the reasons why you'd
>> want 1,
>> > 4, or 16.  As a follow up, if we can get a token simulation written we
>> can
>> > try all sorts of topologies with whatever token algorithms we want.
>> Once
>> > that simulation is written and we've got some reports we can revisit.
>>
>>
>> This works for me, for our first step forward.
>> Good docs will always empower users more than any default setting can!
>>
>> cheers,
>> Mick
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>>

Re: Update defaults for 4.0?

[Jeremy Hanna]

I just wanted to close the loop for
https://issues.apache.org/jira/browse/CASSANDRA-14902 (update
compaction_throughput_mb_per_sec default from 16 to 64) - the default will
become 64 for 4.0 unless people had strong objections.
I'll update the separate discussion on num_tokens (
https://issues.apache.org/jira/browse/CASSANDRA-13701) but that is getting
finalized after some testing.

On Sat, Jan 25, 2020 at 2:59 AM Alexander Dejanovski 
wrote:

> I support changing the default GC settings. The ones we have now drive me
> nuts.
> We should raise the max heap size for CMS to 16G instead of 8 now. We
> should still not go higher than half the available RAM.
> Also, we should set a new gen size between 40% and 50% of the heap size.
> The 100MB per core rule for computing the new gen size doesn't make any
> sense IMO (at least in the context of Cassandra).
>
> This is one of the most common optimizations we make on clusters, and most
> peeps that run Cassandra aren't GC experts (and shouldn't be).
>
> -
> Alexander Dejanovski
> France
> @alexanderdeja
>
> Consultant
> Apache Cassandra Consulting
> http://www.thelastpickle.com
>
>
> On Fri, Jan 24, 2020 at 3:48 PM Joshua McKenzie 
> wrote:
>
> > >
> > > I'm unable to create an epic in the project - not sure if that has to
> do
> > > with project permissions.  Could someone create an epic and link these
> > > tickets as subtasks?
> >
> >
> > Just realized I can no longer create epics anymore (or the "new" JIRA UI
> is
> > just so obtuse I can't figure it out. I give it 50/50 odds). Thinking
> this
> > may have been due to transition to LDAP.
> >
> > Since I planned on experimenting with the whole "what does the confluent
> > testing page look like in an epic" today, I'll ping Nate once it's not
> > godawfully early in NZ about this. Or he'll read this email, either way.
> :)
> >
> > On Thu, Jan 23, 2020 at 11:13 PM Jeremy Hanna <
> jeremy.hanna1...@gmail.com>
> > wrote:
> >
> > > I've previously created
> > > https://issues.apache.org/jira/browse/CASSANDRA-14902 for updating the
> > > compaction_throughput_in_mb default.  I created
> > > https://issues.apache.org/jira/browse/CASSANDRA-15521 for updating the
> > > num_tokens default,
> > https://issues.apache.org/jira/browse/CASSANDRA-15522
> > > for updating the [roles|permissions|credentials]_validity_in_ms
> defaults,
> > > and https://issues.apache.org/jira/browse/CASSANDRA-15523 for updating
> > the
> > > default snitch to GossipingPropertyFileSnitch.
> > > I'm unable to create an epic in the project - not sure if that has to
> do
> > > with project permissions.  Could someone create an epic and link these
> > > tickets as subtasks?
> > > Jon - would you mind creating the ticket around JVM defaults?  Are you
> > > thinking of the default GC and settings for a better out of the box
> > > experience?
> > > Thanks all,
> > > Jeremy
> > >
> > > On Fri, Jan 24, 2020 at 1:57 PM Jon Haddad  wrote:
> > >
> > > > Yes. please do. We should also update our JVM defaults.
> > > >
> > > > On Thu, Jan 23, 2020, 9:28 PM Jeremy Hanna <
> jeremy.hanna1...@gmail.com
> > >
> > > > wrote:
> > > >
> > > > > To summarize this thread, I think people are generally okay with
> > > updating
> > > > > certain defaults for 4.0 provided we make sure it doesn't
> > unpleasantly
> > > > > surprise cluster operators.  I think with the num_tokens and
> > > > > compaction_throughput_in_mb we could go with a release note for the
> > > > reasons
> > > > > in my last email.  I also agree that we should consider bump
> > > > > roles_validity_in_ms, permissions_validity_in_ms, and
> > > > >credentials_validity_in_ms along with the default snitch (going
> to
> > > > GPFS
> > > > > as the default) as that gives people a DC aware default at least to
> > > > start.
> > > > >
> > > > > Is everyone okay if I create tickets for each of these and link
> them
> > > with
> > > > > an epic so that we can discuss them separately?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Jeremy
> > > > >
> > > > > On Thu, Jan 23, 2020 at 5:34 AM Alex Ott 
> wrote:
> > > > >
> > > > > > In addition to t

Moving forward towards our best release yet

[Jeremy Hanna]

I've been in the Cassandra community for about 10 years now and I've seen a lot 
of ups and downs.  I care deeply about both the project and the people 
interacting on the project personally.  I consider many of you to be good 
friends.

Regardless of the history that's caused some friction on recent discussion 
threads, I hope we can all see past the "us versus them" towards shipping 
something excellent and building momentum.

I would just ask - please assume that everyone wants the project to succeed - 
to build the best, most stable, most scalable, most developer and operationally 
friendly database out there.  Please know that while you personally may have 
seen X clusters in your work with Y nodes with Z challenges, you're in good 
company - we all have.  Let's assume that everyone has a unique contribution 
based on battle scars and triumphs.  As we listen to each other with this in 
mind, I think we can move forward more effectively.  There are so many 
complementary efforts that can help make things more stable, reproduce issues 
and test for regressions now.  As we get into the final stages of the 4.0 
release cycle, I think we can bring all of this to bear for the best release 
we've ever had.

We all have different viewpoints but please let's assume the best in others and 
communicate constructively.  We all have things to contribute - large or small 
- and it's great to see renewed interest with new contributors.  With all of 
the energy leading up to the release, I think we're seeing a glimpse of what we 
can do as a revitalized project and community and this is just the beginning.

Thanks for all you do,

Jeremy
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Project governance wiki doc (take 2)

[Jeremy Hanna]

> On Jun 25, 2020, at 10:56 AM, Jordan West  wrote:
> 
> On Wed, Jun 24, 2020 at 3:43 PM Dinesh Joshi  wrote:
> 
>> 3. Discussion #3 - "... 1 business day notice period."  Whose business day
>> is it? US? Europe? Australia? NZ? We are a distributed community and so 1
>> business day is ambiguous. ASF typically states a 48-72 hour period which
>> gives enough time to cover everyone in the community. We want to avoid
>> people getting disenfranchised due to their location. I propose we make
>> this longer and avoid using 'business day' language.
>> 
>> 
> I'll take responsibility for that. It was one of the discussions on the
> google doc during the initial round of feedback.  The intention was to
> ensure folks didn't feel obligated to check the mailing list on the
> weekends or holidays (regardless of location) since we are all volunteering
> our time. I intended it to mean "not on weekends or holidays for you". We
> can use more specific language if we feel its necessary.
> 

I do agree with the intent of not being burdensome for a community of 
volunteers with lives beyond the project.  However I think 48 or 72 hours is 
probably better and left unqualified because it's tricky.  Besides holidays 
which vary wildly, the weekend starts on the US Friday in the Asia Pacific 
region.  Also Friday/Saturday is the weekend in Israel.

> 
>> Thanks,
>> 
>> Dinesh
>> 
>> [1] https://www.apache.org/foundation/voting.html#Veto
>> 
>>> On Jun 24, 2020, at 2:59 PM, sankalp kohli 
>> wrote:
>>> 
>>> +1
>>> 
>>> On Wed, Jun 24, 2020 at 8:37 AM Jake Luciani  wrote:
>>> 
 +1 (b)
 
 On Wed, Jun 24, 2020 at 9:59 AM Joshua McKenzie 
 wrote:
 
> A reminder: this vote will close at midnight PST today in roughly 17
 hours.
> 
> 
> On Mon, Jun 22, 2020 at 2:20 PM J. D. Jordan <
>> jeremiah.jor...@gmail.com>
> wrote:
> 
>> +1 non-binding
>> 
>>> On Jun 22, 2020, at 1:18 PM, Stefan Podkowinski 
> wrote:
>>> 
>>> +1
>>> 
 On 22.06.20 20:12, Blake Eggleston wrote:
 +1
 
>> On Jun 20, 2020, at 8:12 AM, Joshua McKenzie <
 jmcken...@apache.org>
>> wrote:
> 
> Link to doc:
> 
>> 
> 
 
>> https://cwiki.apache.org/confluence/display/CASSANDRA/Apache+Cassandra+Project+Governance
> 
> Change since previous cancelled vote:
> "A simple majority of this electorate becomes the low-watermark for
>> votes
> in favour necessary to pass a motion, with new PMC members added to
> the
> calculation."
> 
> This previously read "super majority". We have lowered the low
 water
>> mark
> to "simple majority" to balance strong consensus against risk of
> stall
>> due
> to low participation.
> 
> 
> - Vote will run through 6/24/20
> - pmc votes considered binding
> - simple majority of binding participants passes the vote
> - committer and community votes considered advisory
> 
> Lastly, I propose we take the count of pmc votes in this thread as
> our
> initial roll call count for electorate numbers and low watermark
> calculation on subsequent votes.
> 
> Thanks again everyone (and specifically Benedict and Jon) for the
> time
>> and
> collaboration on this.
> 
> ~Josh
 
 
 -
 To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
 For additional commands, e-mail: dev-h...@cassandra.apache.org
 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 
> 
 
 
 --
 http://twitter.com/tjake
 
>> 
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [Discuss] num_tokens default in Cassandra 4.0

[Jeremy Hanna]

As discussed, let's go with 16.  Speaking with Anthony privately as well, I
had forgotten that some of the analysis that Branimir had initially done on
the skew and allocation may have been internal to DataStax so I should have
mentioned that previously.  Thanks to Mick, Alex, and Anthony for doing
this analysis and helping back the decision with data.  This will benefit
many that start with Cassandra that don't know that 256 is a bad number and
end up with a hard to change decision later.  I assigned myself to
https://issues.apache.org/jira/browse/CASSANDRA-13701.  Thanks all.

On Wed, Mar 11, 2020 at 6:02 AM Mick Semb Wever  wrote:

>
>
> > I propose we drop it to 16 immediately.  I'll add the production docs
> > in CASSANDRA-15618 with notes on token count, the reasons why you'd want
> 1,
> > 4, or 16.  As a follow up, if we can get a token simulation written we
> can
> > try all sorts of topologies with whatever token algorithms we want.  Once
> > that simulation is written and we've got some reports we can revisit.
>
>
> This works for me, for our first step forward.
> Good docs will always empower users more than any default setting can!
>
> cheers,
> Mick
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: [Discuss] num_tokens default in Cassandra 4.0

[Jeremy Hanna]

I think Mick and Anthony make some valid operational and skew points for 
smaller/starting clusters with 4 num_tokens. There’s an arbitrary line between 
small and large clusters but I think most would agree that most clusters are on 
the small to medium side. (A small nuance is afaict the probabilities have to 
do with quorum on a full token range, ie it has to do with the size of a 
datacenter not the full cluster

As I read this discussion I’m personally more inclined to go with 16 for now. 
It’s true that if we could fix the skew and topology gotchas for those starting 
things up, 4 would be ideal from an availability perspective. However we’re 
still in the brainstorming stage for how to address those challenges. I think 
we should create tickets for those issues and go with 16 for 4.0.

This is about an out of the box experience. It balances availability, 
operations (such as skew and general bootstrap friendliness and 
streaming/repair), and cluster sizing. Balancing all of those, I think for now 
I’m more comfortable with 16 as the default with docs on considerations and 
tickets to unblock 4 as the default for all users.

>>> On Feb 1, 2020, at 6:30 AM, Jeff Jirsa  wrote:
>> On Fri, Jan 31, 2020 at 11:25 AM Joseph Lynch  wrote:
>> I think that we might be bikeshedding this number a bit because it is easy
>> to debate and there is not yet one right answer.
> 
> 
> https://www.youtube.com/watch?v=v465T5u9UKo

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [Discuss] num_tokens default in Cassandra 4.0

[Jeremy Hanna]

The new default wouldn't be retroactively set for 3.x, but the same principles 
apply.  The new algorithm is in 3.x as well as the simplification of the 
configuration.  So no reason not to use the same configuration on 3.x.

> On Jan 30, 2020, at 4:34 AM, Chen-Becker, Derek  
> wrote:
> 
> Does the same guidance apply to 3.x clusters? I read through the JIRA ticket 
> linked below, along with tickets that it links to, but it's not clear that 
> the new allocation algorithm is available in 3.x or if there are other 
> reasons that this would be problematic.
> 
> Thanks,
> 
> Derek
> 
> On 1/29/20, 9:54 AM, "Jon Haddad"  wrote:
> 
>Ive put a lot of my previous clients on 4 tokens, all of which have
>resulted in a major improvement.
> 
>I wouldn't use any more than 4 except under some pretty unusual
>circumstances.
> 
>Jon
> 
>On Wed, Jan 29, 2020, 11:18 AM Ben Bromhead  wrote:
> 
>> +1 to reducing the number of tokens as low as possible for availability
>> issues. 4 lgtm
>> 
>> On Wed, Jan 29, 2020 at 1:14 AM Dinesh Joshi  wrote:
>> 
>>> Thanks for restarting this discussion Jeremy. I personally think 4 is a
>>> good number as a default. I think whatever we pick, we should have enough
>>> documentation for operators to make sense of the new defaults in 4.0.
>>> 
>>> Dinesh
>>> 
>>>> On Jan 28, 2020, at 9:25 PM, Jeremy Hanna 
>>> wrote:
>>>> 
>>>> I wanted to start a discussion about the default for num_tokens that
>>> we'd like for people starting in Cassandra 4.0.  This is for ticket
>>> CASSANDRA-13701 <https://issues.apache.org/jira/browse/CASSANDRA-13701>
>>> (which has been duplicated a number of times, most recently by me).
>>>> 
>>>> TLDR, based on availability concerns, skew concerns, operational
>>> concerns, and based on the fact that the new allocation algorithm can be
>>> configured fairly simply now, this is a proposal to go with 4 as the new
>>> default and the allocate_tokens_for_local_replication_factor set to 3.
>>> That gives a good experience out of the box for people and is the most
>>> conservative.  It does assume that racks and DCs have been configured
>>> correctly.  We would, of course, go into some detail in the NEWS.txt.
>>>> 
>>>> Joey Lynch and Josh Snyder did an extensive analysis of availability
>>> concerns with high num_tokens/virtual nodes in their paper <
>>> 
>> http://mail-archives.apache.org/mod_mbox/cassandra-dev/201804.mbox/%3CCALShVHcz5PixXFO_4bZZZNnKcrpph-=5QmCyb0M=w-mhdyl...@mail.gmail.com%3E
>>> .
>>> This worsens as clusters grow larger.  I won't quote the paper here but
>> in
>>> order to have a conservative default and with the accompanying new
>>> allocation algorithm, I think it makes sense as a default.
>>>> 
>>>> The difficulties have always been that virtual nodes have been
>>> beneficial for operations but that 256 is too high for the purposes of
>>> repair and as Joey and Josh cover, for availability.  Going lower with
>> the
>>> original allocation algorithm has produced skew in allocation in its
>> naive
>>> distribution.  Enter CASSANDRA-7032 <
>>> https://issues.apache.org/jira/browse/CASSANDRA-7032> and the new token
>>> allocation algorithm.  CASSANDRA-15260 <
>>> https://issues.apache.org/jira/browse/CASSANDRA-15260> makes the new
>>> algorithm operationally simpler.
>>>> 
>>>> One other item of note - since Joey and Josh's analysis, there have
>> been
>>> improvements in streaming and other considerations that can reduce the
>>> probability of more than one node representing some token range being
>>> unavailable, but it would still be good to be conservative.
>>>> 
>>>> Please chime in with any concerns with having num_tokens=4 and
>>> allocate_tokens_for_local_replication_factor=3 and the accompanying
>>> rationale so we can improve the experience for all users.
>>>> 
>>>> Other resources:
>>>> 
>>> 
>> https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
>>>> 
>>> 
>> https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/config/configVnodes.html
>>>> 
>>> 
>> https://www.datastax.com/blog/2016/01/new-token-allocation-algorithm-cassandra-30
>>>> 
>>> 
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 
>> --
>> 
>> Ben Bromhead
>> 
>> Instaclustr | www.instaclustr.com | @instaclustr
>> <http://twitter.com/instaclustr> | (650) 284 9692
>> 
> 
> 
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

[Discuss] num_tokens default in Cassandra 4.0

[Jeremy Hanna]

I wanted to start a discussion about the default for num_tokens that we'd like 
for people starting in Cassandra 4.0.  This is for ticket CASSANDRA-13701 
 (which has been 
duplicated a number of times, most recently by me).

TLDR, based on availability concerns, skew concerns, operational concerns, and 
based on the fact that the new allocation algorithm can be configured fairly 
simply now, this is a proposal to go with 4 as the new default and the 
allocate_tokens_for_local_replication_factor set to 3.  That gives a good 
experience out of the box for people and is the most conservative.  It does 
assume that racks and DCs have been configured correctly.  We would, of course, 
go into some detail in the NEWS.txt.

Joey Lynch and Josh Snyder did an extensive analysis of availability concerns 
with high num_tokens/virtual nodes in their paper 
.
  This worsens as clusters grow larger.  I won't quote the paper here but in 
order to have a conservative default and with the accompanying new allocation 
algorithm, I think it makes sense as a default.

The difficulties have always been that virtual nodes have been beneficial for 
operations but that 256 is too high for the purposes of repair and as Joey and 
Josh cover, for availability.  Going lower with the original allocation 
algorithm has produced skew in allocation in its naive distribution.  Enter 
CASSANDRA-7032  and the 
new token allocation algorithm.  CASSANDRA-15260 
 makes the new algorithm 
operationally simpler.

One other item of note - since Joey and Josh's analysis, there have been 
improvements in streaming and other considerations that can reduce the 
probability of more than one node representing some token range being 
unavailable, but it would still be good to be conservative.

Please chime in with any concerns with having num_tokens=4 and 
allocate_tokens_for_local_replication_factor=3 and the accompanying rationale 
so we can improve the experience for all users.

Other resources:
https://thelastpickle.com/blog/2019/02/21/set-up-a-cluster-with-even-token-distribution.html
https://docs.datastax.com/en/dse/6.7/dse-admin/datastax_enterprise/config/configVnodes.html
https://www.datastax.com/blog/2016/01/new-token-allocation-algorithm-cassandra-30

Re: Update defaults for 4.0?

[Jeremy Hanna]

I've previously created
https://issues.apache.org/jira/browse/CASSANDRA-14902 for updating the
compaction_throughput_in_mb default.  I created
https://issues.apache.org/jira/browse/CASSANDRA-15521 for updating the
num_tokens default, https://issues.apache.org/jira/browse/CASSANDRA-15522
for updating the [roles|permissions|credentials]_validity_in_ms defaults,
and https://issues.apache.org/jira/browse/CASSANDRA-15523 for updating the
default snitch to GossipingPropertyFileSnitch.
I'm unable to create an epic in the project - not sure if that has to do
with project permissions.  Could someone create an epic and link these
tickets as subtasks?
Jon - would you mind creating the ticket around JVM defaults?  Are you
thinking of the default GC and settings for a better out of the box
experience?
Thanks all,
Jeremy

On Fri, Jan 24, 2020 at 1:57 PM Jon Haddad  wrote:

> Yes. please do. We should also update our JVM defaults.
>
> On Thu, Jan 23, 2020, 9:28 PM Jeremy Hanna 
> wrote:
>
> > To summarize this thread, I think people are generally okay with updating
> > certain defaults for 4.0 provided we make sure it doesn't unpleasantly
> > surprise cluster operators.  I think with the num_tokens and
> > compaction_throughput_in_mb we could go with a release note for the
> reasons
> > in my last email.  I also agree that we should consider bump
> > roles_validity_in_ms, permissions_validity_in_ms, and
> >credentials_validity_in_ms along with the default snitch (going to
> GPFS
> > as the default) as that gives people a DC aware default at least to
> start.
> >
> > Is everyone okay if I create tickets for each of these and link them with
> > an epic so that we can discuss them separately?
> >
> > Thanks,
> >
> > Jeremy
> >
> > On Thu, Jan 23, 2020 at 5:34 AM Alex Ott  wrote:
> >
> > > In addition to these, maybe we could consider to change other as well?
> > > Like:
> > >
> > > 1. bump roles_validity_in_ms, permissions_validity_in_ms, and
> > >credentials_validity_in_ms as well - maybe at least to a minute, or
> > 2. I
> > >have seen multiple times when authentication was failing under the
> > heavy
> > >load because queries to system tables were timing out - with these
> > >defaults people may still have the possibility to get updates to
> > >roles/credentials faster when specifying _update_interval_ variants
> of
> > >these configurations.
> > > 2. change default snitch from SimpleSnitch to
> > GossipingPropertyFileSnitch -
> > >we're anyway saying that SimpleSnitch is only appropriate for
> > >single-datacenter deployments, and for real production we need to
> use
> > >GossipingPropertyFileSnitch - why not to set it as default?
> > >
> > >
> > > Jeremy Hanna  at "Wed, 22 Jan 2020 11:22:36 +1100" wrote:
> > >  JH> I mentioned this in the contributor meeting as a topic to bring up
> > on
> > > the list - should we
> > >  JH> take the opportunity to update defaults for Cassandra 4.0?
> > >
> > >  JH> The rationale is two-fold:
> > >  JH> 1) There are best practices and tribal knowledge around certain
> > > properties where people
> > >  JH> just know to update those properties immediately as a starting
> > > point.  If it's pretty much
> > >  JH> a given that we set something as a starting point different than
> the
> > > current defaults, why
> > >  JH> not make that the new default?
> > >  JH> 2) We should align the defaults with what we test with.  There may
> > be
> > > exceptions if we
> > >  JH> have one-off tests but on the whole, we should be testing with
> > > defaults.
> > >
> > >  JH> As a starting point, compaction throughput and number of vnodes
> seem
> > > like good candidates
> > >  JH> but it would be great to get feedback for any others.
> > >
> > >  JH> For compaction throughput (
> > > https://jira.apache.org/jira/browse/CASSANDRA-14902), I've made
> > >  JH> a basic case on the ticket to default to 64 just as a starting
> point
> > > because the decision
> > >  JH> for 16 was made when spinning disk was most common.  Hence most
> > > people I know change that
> > >  JH> and I think without too much bikeshedding, 64 is a reasonable
> > > starting point.  A case
> > >  JH> could be made that empirically the compaction throughput throttle
> > may
> > > have less effect
> > >  JH> than ma

Re: Update defaults for 4.0?

[Jeremy Hanna]

To summarize this thread, I think people are generally okay with updating
certain defaults for 4.0 provided we make sure it doesn't unpleasantly
surprise cluster operators.  I think with the num_tokens and
compaction_throughput_in_mb we could go with a release note for the reasons
in my last email.  I also agree that we should consider bump
roles_validity_in_ms, permissions_validity_in_ms, and
   credentials_validity_in_ms along with the default snitch (going to GPFS
as the default) as that gives people a DC aware default at least to start.

Is everyone okay if I create tickets for each of these and link them with
an epic so that we can discuss them separately?

Thanks,

Jeremy

On Thu, Jan 23, 2020 at 5:34 AM Alex Ott  wrote:

> In addition to these, maybe we could consider to change other as well?
> Like:
>
> 1. bump roles_validity_in_ms, permissions_validity_in_ms, and
>credentials_validity_in_ms as well - maybe at least to a minute, or 2. I
>have seen multiple times when authentication was failing under the heavy
>load because queries to system tables were timing out - with these
>defaults people may still have the possibility to get updates to
>roles/credentials faster when specifying _update_interval_ variants of
>these configurations.
> 2. change default snitch from SimpleSnitch to GossipingPropertyFileSnitch -
>we're anyway saying that SimpleSnitch is only appropriate for
>single-datacenter deployments, and for real production we need to use
>GossipingPropertyFileSnitch - why not to set it as default?
>
>
> Jeremy Hanna  at "Wed, 22 Jan 2020 11:22:36 +1100" wrote:
>  JH> I mentioned this in the contributor meeting as a topic to bring up on
> the list - should we
>  JH> take the opportunity to update defaults for Cassandra 4.0?
>
>  JH> The rationale is two-fold:
>  JH> 1) There are best practices and tribal knowledge around certain
> properties where people
>  JH> just know to update those properties immediately as a starting
> point.  If it's pretty much
>  JH> a given that we set something as a starting point different than the
> current defaults, why
>  JH> not make that the new default?
>  JH> 2) We should align the defaults with what we test with.  There may be
> exceptions if we
>  JH> have one-off tests but on the whole, we should be testing with
> defaults.
>
>  JH> As a starting point, compaction throughput and number of vnodes seem
> like good candidates
>  JH> but it would be great to get feedback for any others.
>
>  JH> For compaction throughput (
> https://jira.apache.org/jira/browse/CASSANDRA-14902), I've made
>  JH> a basic case on the ticket to default to 64 just as a starting point
> because the decision
>  JH> for 16 was made when spinning disk was most common.  Hence most
> people I know change that
>  JH> and I think without too much bikeshedding, 64 is a reasonable
> starting point.  A case
>  JH> could be made that empirically the compaction throughput throttle may
> have less effect
>  JH> than many people think, but I still think an updated default would
> make sense.
>
>  JH> For number of vnodes, Michael Shuler made the point in the discussion
> that we already test
>  JH> with 32, which is a far better number than the 256 default.  I know
> many new users that
>  JH> just leave the 256 default and then discover later that it's better
> to go lower.  I think
>  JH> 32 is a good balance.  One could go lower with the new algorithm but
> I think 32 is much
>  JH> better than 256 without being too skewed, and it's what we currently
> test.
>
>  JH> Jeff brought up a good point that we want to be careful with defaults
> since changing them
>  JH> could come as an unpleasant surprise to people who don't explicitly
> set them.  As a
>  JH> general rule, we should always update release notes to clearly state
> that a default has
>  JH> changed.  For these two defaults in particular, I think it's safe.
> For compaction
>  JH> throughput I think a release not is sufficient in case they want to
> modify it.  For number
>  JH> of vnodes, it won't affect existing deployments with data - it would
> be for new clusters,
>  JH> which would honestly benefit from this anyway.
>
>  JH> The other point is whether it's too late to go into 4.0.  For these
> two changes, I think
>  JH> significant testing can still be done with these new defaults before
> release and I think
>  JH> testing more explicitly with 32 vnodes in particular will give people
> more confidence in
>  JH> the lower number with a wider array of testing (where we don't
> already use 32 explicitly).
>
>  JH> In summary, a

Re: Update defaults for 4.0?

[Jeremy Hanna]

I think what he means is that if users have existing clusters with
num_tokens=256 (current default) and the default changes to 32, the node
won't ignore the value, it will fail to start with an error that you cannot
change from one num_tokens value to another:
ERROR [main] 2020-01-22 17:10:53,159 CassandraDaemon.java:759 - Fatal
configuration error
org.apache.cassandra.exceptions.ConfigurationException: Cannot change the
number of tokens from 256 to 32
at
org.apache.cassandra.service.StorageService.joinTokenRing(StorageService.java:1035)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:717)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.service.StorageService.initServer(StorageService.java:651)
~[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.service.CassandraDaemon.setup(CassandraDaemon.java:388)
[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.service.CassandraDaemon.activate(CassandraDaemon.java:620)
[apache-cassandra-3.11.5.jar:3.11.5]
at
org.apache.cassandra.service.CassandraDaemon.main(CassandraDaemon.java:742)
[apache-cassandra-3.11.5.jar:3.11.5]

For that, he's correct.  I was thinking of when you change initial_token in
the yaml.  If you change the initial_token value(s) and there's already
data on disk, it will happily start and just tell you that is using the
saved tokens it already has, thank you very much.  So it doesn't fail to
start, it just ignores the change.  I had thought changing the num_tokens
would do something but I was wrong.

So there are two scenarios for changing the num_tokens default:
1) people need to be made aware of the change because if they try to
upgrade their existing nodes to 4.0+ and they don't change from the
defaults, then it will fail to start based on this change.
2) as Jeff mentions, if they add a new node into their cluster with the new
default and they were previously using 256, then the new node will claim
just 1/8th of the data as the other nodes.

For 1, I would hope especially for all of the new changes in 4.0 that
people would read the release notes and do their due diligence to say "I'm
going to diff my config differences from the version it corresponds to and
then modify the new config as seems appropriate."  If not and it's
different, they'll find out quickly because it will fail fast.  I'm not as
worried about this one.

For 2, it's a good point and again, hopefully people do the same due
diligence when adding new nodes to their clusters on the new version so
that they aren't surprised by the data density.  In a practical sense,
presumably the ops person has already upgraded their cluster to 4.0 before
adding a new node running 4.0 to their cluster.  So if they weren't aware
of the change in the default in the yaml file, they would get that error
mentioned previously and then know for new nodes added to their cluster.
Similarly, num_tokens is explicit by default and set to 256.  So it
shouldn't be a matter of having it commented out in the yaml and it being
whatever the code determines as the default for the common case.  In that
sense, I'm happy that it's not something that changes underneath you
because you don't set it.

So while I agree that the consequence can be severe when adding a new node,
the cluster operator will already be aware of the change when they upgrade
their existing nodes even if they didn't read the release notes or do their
config due diligence or just simply missed it, which happens as well - it's
a big upgrade.  So if that's all there is, I don't think the change will be
disruptive outside a surprise if they hadn't noticed the change where it
fails fast.

On Wed, Jan 22, 2020 at 5:02 PM Jeff Jirsa  wrote:

> On Tue, Jan 21, 2020 at 7:41 PM Jonathan Koppenhofer 
> wrote:
>
> > If someone isn't explicitly setting vnodes, and the default changes, it
> > will vary from the number of assigned tokens for existing clusters,
> right?
> > Won't this cause the node to fail to start?
> >
>
> Nope. You can have 32 tokens on some instances and 256 in other instances
> in the same dc/cluster. No error. The hosts with 256 tokens will just have
> 8x as much data as the hosts with 32 tokens. And that's why changing
> defaults is hard.
>
>
>
> >
> > I am in favor of changing these defaults, but should provide very clear
> > guidance on vnodes (unless I am wrong).
> >
> > I'm sure there are others that would be safe to change. I'll review our
> > defaults we typically set and report back tomorrow.
> >
> > On Tue, Jan 21, 2020, 7:22 PM Jeremy Hanna 
> > wrote:
> >
> > > I mentioned this in the contributor meeting as a topic to bring up on
> the
> > > list - should we take the opportunity to update defaults for Cassandra
> > 4.0?
> > >
> > > The rationale is two-f

Update defaults for 4.0?

[Jeremy Hanna]

I mentioned this in the contributor meeting as a topic to bring up on the list 
- should we take the opportunity to update defaults for Cassandra 4.0?

The rationale is two-fold:
1) There are best practices and tribal knowledge around certain properties 
where people just know to update those properties immediately as a starting 
point.  If it's pretty much a given that we set something as a starting point 
different than the current defaults, why not make that the new default?
2) We should align the defaults with what we test with.  There may be 
exceptions if we have one-off tests but on the whole, we should be testing with 
defaults.

As a starting point, compaction throughput and number of vnodes seem like good 
candidates but it would be great to get feedback for any others.

For compaction throughput 
(https://jira.apache.org/jira/browse/CASSANDRA-14902), I've made a basic case 
on the ticket to default to 64 just as a starting point because the decision 
for 16 was made when spinning disk was most common.  Hence most people I know 
change that and I think without too much bikeshedding, 64 is a reasonable 
starting point.  A case could be made that empirically the compaction 
throughput throttle may have less effect than many people think, but I still 
think an updated default would make sense.

For number of vnodes, Michael Shuler made the point in the discussion that we 
already test with 32, which is a far better number than the 256 default.  I 
know many new users that just leave the 256 default and then discover later 
that it's better to go lower.  I think 32 is a good balance.  One could go 
lower with the new algorithm but I think 32 is much better than 256 without 
being too skewed, and it's what we currently test.

Jeff brought up a good point that we want to be careful with defaults since 
changing them could come as an unpleasant surprise to people who don't 
explicitly set them.  As a general rule, we should always update release notes 
to clearly state that a default has changed.  For these two defaults in 
particular, I think it's safe.  For compaction throughput I think a release not 
is sufficient in case they want to modify it.  For number of vnodes, it won't 
affect existing deployments with data - it would be for new clusters, which 
would honestly benefit from this anyway.

The other point is whether it's too late to go into 4.0.  For these two 
changes, I think significant testing can still be done with these new defaults 
before release and I think testing more explicitly with 32 vnodes in particular 
will give people more confidence in the lower number with a wider array of 
testing (where we don't already use 32 explicitly).

In summary, are people okay with considering updating these defaults and 
possibly others in the alpha stage of a new major release?  Are there other 
properties to consider?

Jeremy
-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Release Apache Cassandra 4.0-alpha1 (24 hour vote)

[Jeremy Hanna]

(non-binding) +1

Speaking of the testing, thanks all for the different tooling and test runs.  
Things like fqltool, cassandra-diff, and the testing that Joey has been doing 
with https://issues.apache.org/jira/browse/CASSANDRA-14747 
 are great.  These will 
all improve the quality of the release.

> On Sep 5, 2019, at 4:03 PM, Joseph Lynch  wrote:
> 
> Running all tests at
> https://circleci.com/workflow-run/79918e2a-ea8e-48a6-a38d-96cf85de27ff
> 
> Will report back with results shortly,
> -Joey
> 
> On Thu, Sep 5, 2019 at 3:55 PM Jon Haddad  wrote:
> 
>> +1
>> 
>> On Thu, Sep 5, 2019 at 3:44 PM Michael Shuler 
>> wrote:
>> 
>>> I propose the following artifacts for release as 4.0-alpha1.
>>> 
>>> sha1: fc4381ca89ab39a82c9018e5171975285cc3bfe7
>>> Git:
>>> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=shortlog;h=refs/tags/4.0-alpha1-tentative
>>> Artifacts:
>>> 
>>> 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1177/org/apache/cassandra/apache-cassandra/4.0-alpha1/
>>> Staging repository:
>>> 
>> https://repository.apache.org/content/repositories/orgapachecassandra-1177/
>>> 
>>> The Debian and RPM packages are available here:
>>> http://people.apache.org/~mshuler
>>> 
>>> The vote will be open for 24 hours (longer if needed).
>>> 
>>> [1]: CHANGES.txt:
>>> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-alpha1-tentative
>>> [2]: NEWS.txt:
>>> 
>>> 
>> https://gitbox.apache.org/repos/asf?p=cassandra.git;a=blob_plain;f=CHANGES.txt;hb=refs/tags/4.0-alpha1-tentative
>>> 
>>> -
>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>> 
>>> 
>> 

Re: Contributing cassandra-diff

[Jeremy Hanna]

It’s great to contribute such a tool. The change between 2.x and 3.0 brought a 
translation layer from thrift to cql that is hard to validate on real clusters 
without something like this. Thank you.

As for naming, perhaps cassandra-compare might be clearer as diff is an 
overloaded word but that’s a bikeshed sort of argument.

> On Aug 22, 2019, at 12:32 AM, Vinay Chella  wrote:
> 
> This is a great addition to our Cassandra validation framework/tools. I can
> see many teams in the community get benefited from tooling like this.
> 
> I like the idea of the generic repo (repos/asf/cassandra-contrib.git
> or *whatever
> the name is*) for tools like this, for the following 2 main reasons.
> 
>   1. Easily accessible/ reachable/ searchable
>   2. Welcomes community in Cassandra ecosystem to contribute more easily
> 
> 
> 
> Thanks,
> Vinay Chella
> 
> 
>> On Wed, Aug 21, 2019 at 11:39 PM Marcus Eriksson  wrote:
>> 
>> Hi, we are about to open source our tooling for comparing two cassandra
>> clusters and want to get some feedback where to push it. I think the
>> options are: (name bike-shedding welcome)
>> 
>> 1. create repos/asf/cassandra-diff.git
>> 2. create a generic repos/asf/cassandra-contrib.git where we can add more
>> contributed tools in the future
>> 
>> Temporary location: https://github.com/krummas/cassandra-diff
>> 
>> Cassandra-diff is a spark job that compares the data in two clusters - it
>> pages through all partitions and reads all rows for those partitions in
>> both clusters to make sure they are identical. Based on the configuration
>> variable “reverse_read_probability” the rows are either read forward or in
>> reverse order.
>> 
>> Our main use case for cassandra-diff has been to set up two identical
>> clusters, transfer a snapshot from the cluster we want to test to these
>> clusters and upgrade one side. When that is done we run this tool to make
>> sure that 2.1 and 3.0 gives the same results. A few examples of the bugs we
>> have found using this tool:
>> 
>> * CASSANDRA-14823: Legacy sstables with range tombstones spanning multiple
>> index blocks create invalid bound sequences on 3.0+
>> * CASSANDRA-14803: Rows that cross index block boundaries can cause
>> incomplete reverse reads in some cases
>> * CASSANDRA-15178: Skipping illegal legacy cells can break reverse
>> iteration of indexed partitions
>> 
>> /Marcus
>> 
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>> 
>> 

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: JIRA Workflow Proposals

[Jeremy Hanna]

Regarding labels, I am personally a fan of both - the mapping of commonly used 
labels to things like components, features, tools, etc. as well as keeping 
labels for newer and more arbitrary groupings.  I’ve tried to maintain certain 
labels like virtual-tables, lcs, lwt, fqltool, etc because there are new things 
(e.g. fqltool and virtual tables) that we don’t immediately make into 
components and it's really nice to group them to see where there might be 
stability or feature specific (thinking virtual tables) items.  I agree that 
arbitrary and misspelled labels make things a bit noisy but as long as we 
strive to use the components/features and do some periodic upkeep of labels.  
By periodic upkeep I mean, converting new labels into components or what have 
you.  Beyond new features or arbitrary groupings, it might have been nice to 
have had ngcc labeled tickets to see how that’s contributed to the project over 
time or some other similar event.

In summary, I really like the mapping but I also really like the way that 
labels can still be of value.  Also, if we strive to keep the components field 
up to date, there’s really no harm in having the labels.



Jeremy

> On Nov 26, 2018, at 8:33 AM, Sankalp Kohli  wrote:
> 
> I have following initial comments on the proposal. 
> 1. Creating a JIRA should have few fields mandatory like platform, version, 
> etc. We want to put less burden on someone creating a ticket but I feel these 
> are required for opening bugs. 
> 
> 2. What is the flow when a non committer does the first pass of review? 
> 
> 
> 
>> On Nov 26, 2018, at 7:46 PM, Joshua McKenzie  wrote:
>> 
>> 1) Removal of labels: one of the strengths of the current model is
>> flexibility for groupings of concepts to arise from a user-perspective
>> (lhf, etc). Calcifying the label concepts into components, categories, etc.
>> is a strict loss of functionality for users to express and categorize their
>> concerns with the project. This feels like an over-correction to our
>> current lack of discipline cleaning up one-off labels. Counter-proposal:
>> 
>>  1. We beef up the categories and components as proposed and migrate
>>  labels to those
>>  2. We remove the one-off labels that aren't adding value, considering
>>  aggregating similar labels
>>  3. We leave the "labels" field intact, perhaps with a bit of guidance on
>>  how to effectively use it
>> 
>> 2) Required fields on transition: assuming these are required upon *issue
>> creation*, that's placing a significant burden on a user that's seen
>> something and wants to open a ticket about it, but isn't sure if it's a
>> "Semantic Failure" or a "Consistency Failure", for instance. If this is,
>> instead, a field required for transition to other ticket states by the
>> developer working on it, I think that makes sense.
>> 
>> 3) Priority Changes: to be blunt, this looks like shuffling chairs on the
>> deck of the titanic on this one - in my experience, most users aren't going
>> to set the priority on tickets when they open them (hence default == major
>> and most tickets are opened as major), so this is something that will be
>> either a) left alone so as not to offend those for whom the priority is
>> *actually* major or consistent with the default, or b) changed by the dev
>> anyway and the added signal from a new "High vs. Urgent" distinction and
>> communication of developer intent to engage with a ticket is something
>> that'll be lost on most users that are just reporting something. I don't
>> have a meaningful counter-proposal at this point as the current problem is
>> more about UX on this field than the signal / noise on the field itself IMO.
>> 
>> A meta question about the "What and Why" of what we're trying to accomplish
>> here: it sounds like what you're looking at is:
>> 
>>  1. to capture more information
>>  2. simplify the data entry
>>  3. nudge people towards more complete and accurate data entry
>>  4. our ability as a project to measure release quality over time and
>>  identify when Cassandra is ready for (or due a) release.
>> 
>> The proposal in its current form makes certain strong inroads in all of
>> those 4 metrics, but I think the major thing missing is the UX of what
>> we're thinking about changing:
>> 
>>  1. Making it easy for / reduce friction for users to report bugs and
>>  requests into the project JIRA (easy to do things right, hard to do things
>>  wrong) (current proposal is a +1 on "do things right", though I'd argue
>>  against it being *easy*)
>>  2. Asking from the users what they can provide about their experience
>>  and issues and no more
>> 
>> Philosophically, are we trying to make it easier for people that are paid
>> FT to work on C*, are we trying to make things easier for *users* of C*,
>> both, neither? Who are we targeting here w/these project changes and what
>> of their issues / problems are we trying to improve?
>> 
>> And lastly: the TLC and management of the JIRA aspects of this 

Re: Deprecating/removing PropertyFileSnitch?

[Jeremy Hanna]

> On Oct 29, 2018, at 11:20 AM, Jeff Jirsa  wrote:
> 
> On Mon, Oct 29, 2018 at 8:35 AM Jeremy Hanna 
> wrote:
> 
>> Re-reading this thread, it sounds like the issue is there are times when a
>> field may go missing in gossip and it hasn’t yet been tracked down.  As
>> Jeremiah says, can we get that into a Jira issue with any contextual
>> information (if there is any)?  However as he says, in theory fields going
>> missing from gossip shouldn’t cause problems for users of GPFS and I don’t
>> believe there have been issues raised in that regard for all of the
>> clusters out there (including Jeff’s comment about it in this thread).
>> Testing that more thoroughly could also be a dependent ticket of
>> deprecating/removing PFS.
>> 
>> 
> The problem with opening a JIRA now is that it'll look just like 13700 and
> the others before it - it'll read something like "status goes missing in
> large clusters" and the very next time we find a gossip bug, we'll mark it
> as fixed, and it may or may not be the only cause of that bug.

I’ve created a Jira that CASSANDRA-10745 requires for completion to thoroughly 
test the GPFS under such conditions.  See CASSANDRA-14856 
<https://issues.apache.org/jira/browse/CASSANDRA-14856>
> 
> 
>> Separately, both Jeff and Sankalp were saying that the fallback was a
>> problem and there was a flurry of tickets back in 2016 that led to the
>> original ticket to deprecate the property file snitch.  However,
>> https://issues.apache.org/jira/browse/CASSANDRA-10745 <
>> https://issues.apache.org/jira/browse/CASSANDRA-10745> discusses what to
>> do when deprecating it.  Would people want the functionality between GPFS
>> completely separate from PFS or would people want a mode to emulate it
>> while using the code for GPFS underneath?
>> 
> 
> Actually, Jeff was guessing that the class of problems that would make you
> want to deprecate PFS is fallback from GPFS to PFS (because beyond that PFS
> is just stupid easy to use and I can't imagine it's causing a lot of
> problems for people who know they're using PFS - yes, if you don't update
> the file, things break, but that's precisely the guarantee of the snitch).

My apologies if I had misrepresented, but I’m glad I checked.

What I was originally saying is that PFS has these sharp edges to it - if you 
don’t sync the files for whatever reason, there are problems.  I saw a case 
recently where a team upgraded their machines in one DC and their addresses 
were new in that DC.  They updated the properties file in the DC where they 
upgraded machines but neglected to update the addresses in the other DC.  In 
that case, the nodes in the other DC saw nodes that didn’t have any 
configuration for them and assigned the default configuration as per the file 
option, which was incorrect.  That caused some difficult to workaround 
problems.  All of this could have been avoided had they been using the GPFS 
instead.

So in order to not invite problems such as this for those new to the project or 
and just because there are going to be times when there will be configuration 
mismatches resulting in this sort of behavior (even with 
https://issues.apache.org/jira/browse/CASSANDRA-12681 
<https://issues.apache.org/jira/browse/CASSANDRA-12681>), I was hoping to get 
consensus on deprecating/removing PFS.

> 
> 
>> 
>> 
>>> On Oct 22, 2018, at 10:33 PM, Jeremiah D Jordan <
>> jeremiah.jor...@gmail.com> wrote:
>>> 
>>> If you guys are still seeing the problem, would be good to have a JIRA
>> written up, as all the ones linked were fixed in 2017 and 2015.
>> CASSANDRA-13700 was found during our testing, and we haven’t seen any other
>> issues since fixing it.
>>> 
>>> -Jeremiah
>>> 
>>>> On Oct 22, 2018, at 10:12 PM, Sankalp Kohli 
>> wrote:
>>>> 
>>>> No worries...I mentioned the issue not the JIRA number
>>>> 
>>>>> On Oct 22, 2018, at 8:01 PM, Jeremiah D Jordan 
>> wrote:
>>>>> 
>>>>> Sorry, maybe my spam filter got them or something, but I have never
>> seen a JIRA number mentioned in the thread before this one.  Just looked
>> back through again to make sure, and this is the first email I have with
>> one.
>>>>> 
>>>>> -Jeremiah
>>>>> 
>>>>>> On Oct 22, 2018, at 9:37 PM, sankalp kohli 
>> wrote:
>>>>>> 
>>>>>> Here are some of the JIRAs which are fixed but actually did not fix
>> the
>>>>>> issue. We have tried fixing this by several patches. May be it will be
>>>>>> fixed when 

Re: Deprecating/removing PropertyFileSnitch?

[Jeremy Hanna]

>> with GPFS the issue being talked about is predicated on some theoretical
>>>>> gossip bug happening.
>>>>>> In the past year at DataStax we have done a lot of testing on 3.0 and
>>>>> 3.11 around adding nodes, adding DC’s, replacing nodes, replacing racks,
>>>>> and replacing DC’s, all while using GPFS, and as far as I know we have not
>>>>> seen any “lost” rack/DC information during such testing.
>>>>>> 
>>>>>> -Jeremiah
>>>>>> 
>>>>>>> On Oct 22, 2018, at 5:46 PM, sankalp kohli 
>>>>> wrote:
>>>>>>> 
>>>>>>> We will have similar issues with Gossip but this will create more
>>>>> issues as
>>>>>>> more things will be relied on Gossip.
>>>>>>> 
>>>>>>> I agree PFS should be removed but I dont see how it can be with issues
>>>>> like
>>>>>>> these or someone proves that it wont cause any issues.
>>>>>>> 
>>>>>>> On Mon, Oct 22, 2018 at 2:21 PM Paulo Motta 
>>>>>>> wrote:
>>>>>>> 
>>>>>>>> I can understand keeping PFS for historical/compatibility reasons, but
>>>>> if
>>>>>>>> gossip is broken I think you will have similar ring view problems
>>>>> during
>>>>>>>> replace/bootstrap that would still occur with the use of PFS (such as
>>>>>>>> missing tokens, since those are propagated via gossip), so that doesn't
>>>>>>>> seem like a strong reason to keep it around.
>>>>>>>> 
>>>>>>>> With PFS it's pretty easy to shoot yourself in the foot if you're not
>>>>>>>> careful enough to have identical files across nodes and updating it
>>>>> when
>>>>>>>> adding nodes/dcs, so it's seems to be less foolproof than other
>>>>> snitches.
>>>>>>>> While the rejection of verbs to invalid replicas on trunk could address
>>>>>>>> concerns raised by Jeremy, this would only happen after the new node
>>>>> joins
>>>>>>>> the ring, so you would need to re-bootstrap the node and lose all the
>>>>> work
>>>>>>>> done in the original bootstrap.
>>>>>>>> 
>>>>>>>> Perhaps one good reason to use PFS is the ability to easily package it
>>>>>>>> across multiple nodes, as pointed out by Sean Durity on CASSANDRA-10745
>>>>>>>> (which is also it's Achilles' heel). To keep this ability, we could
>>>>> make
>>>>>>>> GPFS compatible with the cassandra-topology.properties file, but
>>>>> reading
>>>>>>>> only the dc/rack info about the local node.
>>>>>>>> 
>>>>>>>> Em seg, 22 de out de 2018 às 16:58, sankalp kohli <
>>>>> kohlisank...@gmail.com>
>>>>>>>> escreveu:
>>>>>>>> 
>>>>>>>>> Yes it will happen. I am worried that same way DC or rack info can go
>>>>>>>>> missing.
>>>>>>>>> 
>>>>>>>>> On Mon, Oct 22, 2018 at 12:52 PM Paulo Motta <
>>>>> pauloricard...@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>> 
>>>>>>>>>>> the new host won’t learn about the host whose status is missing and
>>>>>>>> the
>>>>>>>>>> view of this host will be wrong.
>>>>>>>>>> 
>>>>>>>>>> Won't this happen even with PropertyFileSnitch as the token(s) for
>>>>> this
>>>>>>>>>> host will be missing from gossip/system.peers?
>>>>>>>>>> 
>>>>>>>>>> Em sáb, 20 de out de 2018 às 00:34, Sankalp Kohli <
>>>>>>>>> kohlisank...@gmail.com>
>>>>>>>>>> escreveu:
>>>>>>>>>> 
>>>>>>>>>>> Say you restarted all instances in the cluster and status for some
>>>>>>>> host
>>>>>>>>>>> goes missing. Now when you start a host replacement, the new host
>>>>>>>> won’t
>>>>

Re: [DISCUSS] changing default token behavior for 4.0

[Jeremy Hanna]

I agree that it should be lowered. What I’ve seen debated a bit in the past is 
the number but I don’t think anyone thinks that it should remain 256.

> On Sep 21, 2018, at 7:05 PM, Jonathan Haddad  wrote:
> 
> One thing that's really, really bothered me for a while is how we default
> to 256 tokens still.  There's no experienced operator that leaves it as is
> at this point, meaning the only people using 256 are the poor folks that
> just got started using C*.  I've worked with over a hundred clusters in the
> last couple years, and I think I only worked with one that had lowered it
> to something else.
> 
> I think it's time we changed the default to 4 (or 8, up for debate).
> 
> To improve the behavior, we need to change a couple other things.  The
> allocate_tokens_for_keyspace setting is... odd.  It requires you have a
> keyspace already created, which doesn't help on new clusters.  What I'd
> like to do is add a new setting, allocate_tokens_for_rf, and set it to 3 by
> default.
> 
> To handle clusters that are already using 256 tokens, we could prevent the
> new node from joining unless a -D flag is set to explicitly allow
> imbalanced tokens.
> 
> We've agreed to a trunk freeze, but I feel like this is important enough
> (and pretty trivial) to do now.  I'd also personally characterize this as a
> bug fix since 256 is horribly broken when the cluster gets to any
> reasonable size, but maybe I'm alone there.
> 
> I honestly can't think of a use case where random tokens is a good choice
> anymore, so I'd be fine / ecstatic with removing it completely and
> requiring either allocate_tokens_for_keyspace (for existing clusters)
> or allocate_tokens_for_rf
> to be set.
> 
> Thoughts?  Objections?
> -- 
> Jon Haddad
> http://www.rustyrazorblade.com
> twitter: rustyrazorblade

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: [VOTE] Accept GoCQL driver donation and begin incubation process

[Jeremy Hanna]

I don’t know if others have this same question, but what does accepting the 
gocql driver donation mean?  It becomes a full Apache project separate from 
Cassandra and there exists a separate set of PMC members and such?  Or does it 
become part of the Cassandra project itself?  From Sylvain and Jon’s responses, 
it seems like it’s the latter.  I have some memories of the Apache Extras and 
some things that lived in there for a time and those were eventually phased out 
so I didn’t know if that applied to this discussion as well.

> On Sep 12, 2018, at 10:12 AM, Nate McCall  wrote:
> 
> This will be the same process used for dtest. We will need to walk
> this through the incubator per the process outlined here:
> 
> https://incubator.apache.org/guides/ip_clearance.html
> 
> Pending the outcome of this vote, we will create the JIRA issues for
> tracking and after we go through the process, and discuss adding
> committers in a separate thread (we need to do this atomically anyway
> per general ASF committer adding processes).
> 
> Thanks,
> -Nate
> 
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
> 


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Testing 4.0 Post-Freeze

[Jeremy Hanna]

What I’m saying is that for new contributors, not branching has a cost and I 
think there are other ways to organize efforts.

One of the suggestions was for each organization to test their own use cases in 
the stabilization process.  I don’t think that’s been done very effectively 
previously.  Most organizations only do any sort of significant testing when 
they’re about to put their clusters on the line - i.e. once a version has 
several post GA point releases.  That’s logical and no one wants to be the 
first one to production.  However, if people were to agree to do some form of 
testing pre-release, then I think that would be a step in the right direction.  
In the early days of the project I tried to do this in a small way by testing 
the Hadoop support for every release (major and minor) since it wasn’t on 
everyone’s radar but was important to me.  Then I would vote with a non-binding 
vote and described what was tested.

> On Jul 9, 2018, at 1:30 PM, sankalp kohli  wrote:
> 
> We have done all this for previous releases and we know it has not worked
> well. So how giving it one more try is going to help here. Can someone
> outline what will change for 4.0 which will make it more successful?
> 
> Not branching is one idea but we should discuss what will change now than
> iterating what has already happened.
> 
> On Mon, Jul 9, 2018 at 11:25 AM Jeremy Hanna 
> wrote:
> 
>> I wholeheartedly agree with efforts to make releases more stable and the
>> more contributors that add tests or test within the context of their own
>> use cases, the better.  +1 non-binding to the idea of better, more complete
>> testing for releases.
>> 
>> When it comes to how to do this, I think that’s where there is
>> disagreement.  I personally don’t think that branching detracts from the
>> goal of stabilization.  There are a couple of personas involved here:
>> 
>> * Longtime contributors/committers: I think it’s sufficient to generally
>> ask that whatever time/effort they can contribute be towards stabilizing,
>> testing, and especially testing their production scenarios to cover more
>> surface area when it comes to usage edge cases.  That along with testing
>> longer running things in those scenarios for other types of edge cases.
>> 
>> * New contributors: They likely won’t know about the strategy.  They’re
>> just poking around and looking at lhf tickets or tickets that they need
>> done.  Those are typically low risk but at the same time we don’t want to
>> compromise stability by merging those in.  Reviewing low risk stuff for
>> inclusion doesn’t take a ton of time and gives them a sense that they can
>> be of help and empowers them.  After they have that first experience, then
>> a longer term contributor could share with them a blog post or tldr thread
>> summary about the 4.0 stabilization efforts (would be nice to have one to
>> point people too once we're done).  We could also start creating lhf
>> tickets with stabilization and testing in mind.
>> 
>> Unless we want to make a fundamental change to the project’s process
>> (making trunk stable at all times going forward), then I don’t think
>> there’s a reason why branching would detract from these efforts.  In other
>> words if we’re just trying to say that we want to focus on stabilization
>> for the alpha/beta/rc of 4.0, then I would prefer branching along with
>> clear messaging.
>> 
>> Jeremy
>> 
>>> On Jul 9, 2018, at 12:43 PM, sankalp kohli 
>> wrote:
>>> 
>>> How is this preventing someone from working and collaborating on an
>> Apache
>>> Project? You can still collaborate on making 4.0 more stable.
>>> 
>>> Why will these committers not work on making 4.0 more stable and fix
>> broken
>>> tests? Considering  we cannot release without test passing, how will
>> these
>>> features be used?
>>> 
>>> On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad 
>> wrote:
>>> 
>>>> I don't see how changing the process and banning feature commits is
>>>> going to be any help to the project. There may be a couple committers
>>>> who are interested in committing new features.
>>>> 
>>>> I'm a -1 on changing the branching strategy in a way that prevents
>>>> people from working and collaborating on an Apache project.
>>>> 
>>>> On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
>>>> wrote:
>>>>> 
>>>>> I did not see a -1 but all +0s and a few +1s.
>>>>> 
>>>>> On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
>>&

Re: Testing 4.0 Post-Freeze

[Jeremy Hanna]

I wholeheartedly agree with efforts to make releases more stable and the more 
contributors that add tests or test within the context of their own use cases, 
the better.  +1 non-binding to the idea of better, more complete testing for 
releases.

When it comes to how to do this, I think that’s where there is disagreement.  I 
personally don’t think that branching detracts from the goal of stabilization.  
There are a couple of personas involved here:

* Longtime contributors/committers: I think it’s sufficient to generally ask 
that whatever time/effort they can contribute be towards stabilizing, testing, 
and especially testing their production scenarios to cover more surface area 
when it comes to usage edge cases.  That along with testing longer running 
things in those scenarios for other types of edge cases.

* New contributors: They likely won’t know about the strategy.  They’re just 
poking around and looking at lhf tickets or tickets that they need done.  Those 
are typically low risk but at the same time we don’t want to compromise 
stability by merging those in.  Reviewing low risk stuff for inclusion doesn’t 
take a ton of time and gives them a sense that they can be of help and empowers 
them.  After they have that first experience, then a longer term contributor 
could share with them a blog post or tldr thread summary about the 4.0 
stabilization efforts (would be nice to have one to point people too once we're 
done).  We could also start creating lhf tickets with stabilization and testing 
in mind.

Unless we want to make a fundamental change to the project’s process (making 
trunk stable at all times going forward), then I don’t think there’s a reason 
why branching would detract from these efforts.  In other words if we’re just 
trying to say that we want to focus on stabilization for the alpha/beta/rc of 
4.0, then I would prefer branching along with clear messaging.

Jeremy

> On Jul 9, 2018, at 12:43 PM, sankalp kohli  wrote:
> 
> How is this preventing someone from working and collaborating on an Apache
> Project? You can still collaborate on making 4.0 more stable.
> 
> Why will these committers not work on making 4.0 more stable and fix broken
> tests? Considering  we cannot release without test passing, how will these
> features be used?
> 
> On Mon, Jul 9, 2018 at 10:03 AM Jonathan Haddad  wrote:
> 
>> I don't see how changing the process and banning feature commits is
>> going to be any help to the project. There may be a couple committers
>> who are interested in committing new features.
>> 
>> I'm a -1 on changing the branching strategy in a way that prevents
>> people from working and collaborating on an Apache project.
>> 
>> On Mon, Jul 9, 2018 at 9:56 AM sankalp kohli 
>> wrote:
>>> 
>>> I did not see a -1 but all +0s and a few +1s.
>>> 
>>> On Mon, Jul 9, 2018 at 5:49 AM Josh McKenzie 
>> wrote:
>>> 
 What did we land on? Sounds like we're pretty split without consensus
>> on
 "change the old branching strategy and reject new things on trunk
>> during
 stabilization" vs. "keep doing things the way we did but message
>> strongly
 that we won't be reviewing new things until 4.0 is stable".
 
 On Sat, Jul 7, 2018 at 5:01 PM Sankalp Kohli 
 wrote:
 
> Does anyone has any more feedback on this?
> 
>> On Jul 4, 2018, at 05:36, Aleksey Yeshchenko 
 wrote:
>> 
>> For what it’s worth, I’m fine with both formal branching-level
>> freeze
> and informal ‘let people commit to trunk if they can find a
>> reviewer’ one
> and will support either.
>> 
>> So long as either/both are communicated to the contributors, the
>> only
> difference is in where new feature work gets accumulated: will stay
>> a bit
> longer in personal branches in the latter way.
>> 
>> —
>> AY
>> 
>> On 4 July 2018 at 01:30:40, sankalp kohli (kohlisank...@gmail.com)
> wrote:
>> 
>> Having an explicit way to tell the community that we all will work
>> on
>> testing is better than writing a patch which will sit without
>> review
> for
>> months. I think not having your patch reviewed for months is more
>> discouraging than following the community and helping with
>> stability of
>> 4.0.
>> 
>> 
>> 
>> On Tue, Jul 3, 2018 at 3:02 PM Josh McKenzie >> 
> wrote:
>> 
 
 We propose that between the September freeze date and beta, a new
> branch
 would not be created and trunk would only have bug fixes and
> performance
 improvements committed to it.
>>> 
>>> 
>>> This is more of a call to action and announcement of intent than
>> an
> attempt
 to
 enforce policy; we can and probably will branch off 4.0, and keep
> trunk
 technically active.
>>> 
>>> These are two very different statements. :) Which is it?
>>> 
>>> On Tue, Jul 3, 2018 at 5:57 PM Aleksey Yeshchenko <
>> 

Re: Improve the performance of CAS

[Jeremy Hanna]

Hi Dikang,

Have you seen Blake’s work on implementing egalitarian paxos or epaxos*?  That 
might be helpful for the discussion.

Jeremy

* https://issues.apache.org/jira/browse/CASSANDRA-6246

> On May 16, 2018, at 3:37 PM, Dikang Gu  wrote:
> 
> Hello C* developers,
> 
> I'm working on some performance improvements of the lightweight transitions
> (compare and set), I'd like to hear your thoughts about it.
> 
> As you know, current CAS requires 4 round trips to finish, which is not
> efficient, especially in cross DC case.
> 1) Prepare
> 2) Quorum read current value
> 3) Propose new value
> 4) Commit
> 
> I'm proposing the following improvements to reduce it to 2 round trips,
> which is:
> 1) Combine prepare and quorum read together, use only one round trip to
> decide the ballot and also piggyback the current value in response.
> 2) Propose new value, and then send out the commit request asynchronously,
> so client will not wait for the ack of the commit. In case of commit
> failures, we should still have chance to retry/repair it through hints or
> following read/cas events.
> 
> After the improvement, we should be able to finish the CAS operation using
> 2 rounds trips. There can be following improvements as well, and this can
> be a start point.
> 
> What do you think? Did I miss anything?
> 
> Thanks
> Dikang


-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: Roadmap for 4.0

[Jeremy Hanna]

On Wed, Apr 4, 2018 at 7:50 PM, Michael Shuler 
wrote:

> On 04/04/2018 07:06 PM, Nate McCall wrote:
> >
> > It feels to me like we are coalescing on two points:
> > 1. June 1 as a freeze for alpha
> > 2. "Stable" is the new "Exciting" (and the testing and dogfooding
> > implied by such before a GA)
> >
> > How do folks feel about the above points?
>
> +1
> +1
>

+1 though we may need some additional work on counters ;)


> :)
> Michael
>
> -
> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail: dev-h...@cassandra.apache.org
>
>

Re: Proposal to retroactively mark materialized views experimental

[Jeremy Hanna]

At the risk of sounding redundant it sounds like for MVs at this point we want 
to preserve current functionality for existing users.  We would have a flag in 
the yaml to disable it by default for new MV creation with an error message. In 
addition we want a warning in the log and in cqlsh upon creation and usage even 
with it enabled.  We could be very explicit with a message to the user list and 
a clear NEWS item and upgrade note.

For SASI, it seems like a similar pattern could be used.

For either or both of these features, a Jira epic could be created to make a 
path to making it a full-fledged non-experimental feature.  That could include 
a ticket for realistic testing within the boundaries of the target use cases - 
that way people can point to what testing has been done and what the target use 
cases were.  An epic and test ticket would give people a pathway to make it no 
longer experimental so it isn’t lingering halfway in the codebase indefinitely 
and for interested parties something to contribute towards.  Granted if issues 
are found, they’d need to be added to the epic.

>> On Oct 2, 2017, at 6:16 PM, Mick Semb Wever  wrote:
>> 
>> On 3 October 2017 at 04:57, Aleksey Yeshchenko  wrote:
>> 
>> The idea is to check the flag in CreateViewStatement, so creation of new
>> MVs doesn’t succeed without that flag flipped.
>> Obviously, just disabling existing MVs working in a minor would be silly.
>> As for the warning - yes, that should also be emitted. Unconditionally.
> 
> 
> Thanks Aleksey, this was the best read in this thread imo of what should be
> done with MVs.
> (With these warnings being emitted in both logs and cqlsh).
> 
> Hopefully similar "creation flags" and log+cqlsh warnings can be added to:
> triggers, SASI, and incremental repair (<4.0).
> 
> CDC sounds like it is in the same basket, but it already has the
> `cdc_enabled` yaml flag which defaults false.
> 
> Mick

-
To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
For additional commands, e-mail: dev-h...@cassandra.apache.org

Re: unsubscribe

[Jeremy Hanna]

To unsubscribe, send an email to dev-unsubscr...@cassandra.apache.org 
.
Replying to this email does not unsubscribe you from any list.  It simply 
perpetuates a long thread of useless emails.

> On Apr 10, 2017, at 9:30 AM, Asanka Samaraweera  wrote:
> 
> Unsubscribe 
> 
> -Original Message-
> From: Aleksey Yeschenko [mailto:alek...@apache.org 
> ] 
> Sent: Monday, April 10, 2017 8:29 AM
> To: dev@cassandra.apache.org 
> Subject: Re: unsubscribe
> 
> Unsubscribe applications rejected, sorry.
> 
> -- 
> AY
> 
> On 10 April 2017 at 14:10:05, Sara Prokop (sara.pro...@g2-ops.com) wrote:
> 
> Unsubscribe  
> 
> 
> Sara Prokop  
> G2 Ops, Inc.  
> 
> Office: 757-330-0372  
> 
> 205 Business Park Drive, Suite 200  
> Virginia Beach, VA, 23462  
> 
> 
> This email and any files transmitted with it are confidential and intended 
> solely for the use of the individual or entity to whom they are addressed. If 
> you have received this email in error please notify the system manager. 
> Please note that any views or opinions presented in this email are solely 
> those of the author and do not necessarily represent those of the company. 
> Finally, the recipient should check this email and any attachments for the 
> presence of viruses. The company accepts no liability for any damage caused 
> by any virus transmitted by this email  
> 
>   
> From: alivesubsta...@gmail.com   
> Sent: Sunday, April 9, 2017 9:20:50 AM  
> To: dev@cassandra.apache.org  
> Subject: Re: unsubscribe  
> 
> Unsubscribe  
> 
> Sent from my HTC  
> 
> - Reply message -  
> From: "Michael Kjellman"   
> To: "dev@cassandra.apache.org"   
> Subject: unsubscribe  
> Date: Thu, Apr 6, 2017 19:59  
> 
> 
> https://na01.safelinks.protection.outlook.com/?url=http%3A%2F%2Fapache.org%2Ffoundation%2Fmailinglists.html=02%7C01%7Casamaraweera%40pros.com%7Cf74a41d8751b41e9291c08d4801583e0%7C094cfb7ad13146379047e339e7d04359%7C1%7C0%7C636274277272757970=Qs9S95xCFgxxROvboeqxVmsyhWGNEH50Y%2Fu42TEgKVs%3D=0
>  
> 
>   
> 
> Sent from my iPhone  
> 
> On Apr 6, 2017, at 9:57 AM, Nitija Patil 
> > wrote:  
> 
> unsubscribe  
> 
> On Thu, Apr 6, 2017 at 10:25 PM, Vineet Gadodia 
> > wrote:  
> 
> unsubscribe  
> 
> On Wed, Apr 5, 2017 at 1:51 AM, Ksawery Glab 
> >  
> wrote:  
> 
> unsubscribe  
> 
> 2017-04-05 9:45 GMT+01:00 Nitija Patil 
> >:  
> 
> unsubscribe  
> 
> On Wed, Apr 5, 2017 at 2:05 PM, 郑蒙家(蒙家)  com  
> 
> wrote:  
> 
> unsubscribe  

Re: [VOTE] self-assignment of jira tickets

[Jeremy Hanna]

+1 non-binding.  I’ve always wondered why that restriction was in place.

> On Mar 29, 2017, at 8:21 AM, Jason Brown  wrote:
> 
> Hey all,
> 
> Following up my thread from a week or two ago (
> https://lists.apache.org/thread.html/0665f40c7213654e99817141972c003a2131aba7a1c63d6765db75c5@%3Cdev.cassandra.apache.org%3E),
> I'd like to propose a vote to change to allow any potential contributor to
> assign a jira to themselves without needing to be added to the contributors
> group first.
> 
> https://issues.apache.org/jira/browse/INFRA-11950 is an example of how to
> get this done with INFRA.
> 
> Vote will be open for 72 hours.
> 
> Thanks,
> 
> -Jason Brown

Re: Code quality, principles and rules

[Jeremy Hanna]

https://issues.apache.org/jira/browse/CASSANDRA-7837 may be some interesting 
context regarding what's been worked on to get rid of singletons and static 
initialization.

> On Mar 17, 2017, at 4:47 PM, Jonathan Haddad  wrote:
> 
> I'd like to think that if someone refactors existing code, making it more
> testable (with tests, of course) it should be acceptable on it's own
> merit.  In fact, in my opinion it sometimes makes more sense to do these
> types of refactorings for the sole purpose of improving stability and
> testability as opposed to mixing them with features.
> 
> You referenced the issue I fixed in one of the early emails.  The fix
> itself was a couple lines of code.  Refactoring the codebase to make it
> testable would have been a huge effort.  I wish I had time to do it.  I
> created CASSANDRA-13007 as a follow up with the intent of working on
> compaction from a purely architectural standpoint.  I think this type of
> thing should be done throughout the codebase.
> 
> Removing the singletons is a good first step, my vote is we just rip off
> the bandaid, do it, and move forward.
> 
> On Fri, Mar 17, 2017 at 2:20 PM Edward Capriolo 
> wrote:
> 
>>> On Fri, Mar 17, 2017 at 2:31 PM, Jason Brown  wrote:
>>> 
>>> To François's point about code coverage for new code, I think this makes
>> a
>>> lot of sense wrt large features (like the current work on
>> 8457/12229/9754).
>>> It's much simpler to (mentally, at least) isolate those changed sections
>>> and it'll show up better in a code coverage report. With small patches,
>>> that might be harder to achieve - however, as the patch should come with
>>> *some* tests (unless it's a truly trivial patch), it might just work
>> itself
>>> out.
>>> 
>>> On Fri, Mar 17, 2017 at 11:19 AM, Jason Brown 
>>> wrote:
>>> 
 As someone who spent a lot of time looking at the singletons topic in
>> the
 past, Blake brings a great perspective here. Figuring out and
>>> communicating
 how best to test with the system we have (and of course incrementally
 making that system easier to work with/test) seems like an achievable
>>> goal.
 
 On Fri, Mar 17, 2017 at 10:17 AM, Edward Capriolo <
>> edlinuxg...@gmail.com
 
 wrote:
 
> On Fri, Mar 17, 2017 at 12:33 PM, Blake Eggleston <
>> beggles...@apple.com
 
> wrote:
> 
>> I think we’re getting a little ahead of ourselves talking about DI
>> frameworks. Before that even becomes something worth talking about,
>>> we’d
>> need to have made serious progress on un-spaghettifying Cassandra in
>>> the
>> first place. It’s an extremely tall order. Adding a DI framework
>> right
> now
>> would be like throwing gasoline on a raging tire fire.
>> 
>> Removing singletons seems to come up every 6-12 months, and usually
>> abandoned once people figure out how difficult they are to remove
> properly.
>> I do think removing them *should* be a long term goal, but we really
> need
>> something more immediately actionable. Otherwise, nothing’s going to
>> happen, and we’ll be having this discussion again in a year or so
>> when
>> everyone’s angry that Cassandra 5.0 still isn’t ready for
>> production,
>>> a
>> year after it’s release.
>> 
>> That said, the reason singletons regularly get brought up is because
> doing
>> extensive testing of anything in Cassandra is pretty much
>> impossible,
> since
>> the code is basically this big web of interconnected global state.
> Testing
>> anything in isolation can’t be done, which, for a distributed
>>> database,
> is
>> crazy. It’s a chronic problem that handicaps our ability to release
>> a
>> stable database.
>> 
>> At this point, I think a more pragmatic approach would be to draft
>> and
>> enforce some coding standards that can be applied in day to day
> development
>> that drive incremental improvement of the testing and testability of
>>> the
>> project. What should be tested, how it should be tested. How to
>> write
> new
>> code that talks to the rest of Cassandra and is testable. How to fix
> bugs
>> in old code in a way that’s testable. We should also have some
> guidelines
>> around refactoring the wildly untested sections, how to get started,
> what
>> to do, what not to do, etc.
>> 
>> Thoughts?
> 
> 
> To make the conversation practical. There is one class I personally
>>> really
> want to refactor so it can be tested:
> 
> https://github.com/apache/cassandra/blob/trunk/src/java/org/
> apache/cassandra/net/OutboundTcpConnection.java
> 
> There is little coverage here. Questions like:
> what errors cause the connection to restart?
> when are undropable messages are dropped?
> what happens when the queue fills up?
> Infamous 

Re: Contribute to the Cassandra wiki

[Jeremy Hanna]

The moinmoin wiki was preferred but because of spam, images couldn’t be 
attached.  The options were to use confluence or have a moderated list of 
individuals be approved to update the wiki.  The decision was made to go with 
the latter because of the preference to stick with moinmoin rather than 
confluence.  That’s my understanding of the history there.  I don’t know if 
people would like to revisit using one or the other at this point, though it 
would take a bit of work to convert.

> On Mar 13, 2017, at 9:42 AM, Nate McCall  wrote:
> 
>> Isn't there a way to split tech docs (aka reference) and more
>> user-generated and use-case related/content oriented docs? And maybe to use
>> a more modern WIKI software or scheme. The CS wiki looks like 1998.
> 
> The wiki is what ASF Infra provides by default. Agree that it is a bit
> "old-school."
> 
> I'll ask around about what other projects are doing (or folks who are
> involved in other ASF projects, please chime in).

Re: Contribute to the Cassandra wiki

[Jeremy Hanna]

The moinmoin wiki was preferred but because of spam, images couldn’t be 
attached.  The options were to use confluence or have a moderated list of 
individuals be approved to update the wiki.  The decision was made to go with 
the latter because of the preference to stick with moinmoin rather than 
confluence.  That’s my understanding of the history there.  I don’t know if 
people would like to revisit using one or the other at this point, though it 
would take a bit of work to convert.

> On Mar 13, 2017, at 9:42 AM, Nate McCall  wrote:
> 
>> Isn't there a way to split tech docs (aka reference) and more
>> user-generated and use-case related/content oriented docs? And maybe to use
>> a more modern WIKI software or scheme. The CS wiki looks like 1998.
> 
> The wiki is what ASF Infra provides by default. Agree that it is a bit
> "old-school."
> 
> I'll ask around about what other projects are doing (or folks who are
> involved in other ASF projects, please chime in).

Re: Truncate operation not available in Mutation Object

[Jeremy Hanna]

thanks Chris

> On Feb 23, 2017, at 11:00 AM, Chris Lohfink <chris.lohf...@datastax.com> 
> wrote:
> 
> The truncates are written to the truncated_at field in system.local and
> should be honored by the commit log replayer (
> https://github.com/apache/cassandra/blob/af3fe39dcabd9ef77a00309ce6741268423206df/src/java/org/apache/cassandra/db/commitlog/CommitLogReplayer.java#L102
> ).
> 
> Chris
> 
> On Wed, Feb 22, 2017 at 7:02 PM, Sanal Vasudevan <get2sa...@gmail.com>
> wrote:
> 
>> Thanks Jeremy.
>> Any way I could detect that such a truncate operation was performed on the
>> table? Does it leave a trace that the truncate happened anywhere?
>> 
>> 
>> Best regards,
>> Sanal
>> 
>> On Thu, Feb 23, 2017 at 11:47 AM, Jeremy Hanna <jeremy.hanna1...@gmail.com
>>> 
>> wrote:
>> 
>>> Everything in that table is deleted. There's no mutation or anything in
>>> the commitlog. It's a deletion of all the sstables for that table. To
>> make
>>> sure everything is gone, it first does a flush, then a snapshot to
>> protect
>>> against a mistake, then the truncate itself.
>>> 
>>>> On Feb 22, 2017, at 6:05 PM, Sanal Vasudevan <get2sa...@gmail.com>
>>> wrote:
>>>> 
>>>> Hi Folks,
>>>> 
>>>> I am trying to read Mutations from commit log files through an
>>>> implementation of CommitLogReadHandler interface.
>>>> 
>>>> For a truncate CQL operation, I do not see a Mutation object.
>>>> 
>>>> Does C* skip writing the truncate operation into the commit log file?
>>>> 
>>>> 
>>>> Thanks for your help.
>>>> 
>>>> Best regards,
>>>> Sanal
>>> 
>> 
>> 
>> 
>> --
>> Sanal Vasudevan Nair
>> 

Re: Truncate operation not available in Mutation Object

[Jeremy Hanna]

Everything in that table is deleted. There's no mutation or anything in the 
commitlog. It's a deletion of all the sstables for that table. To make sure 
everything is gone, it first does a flush, then a snapshot to protect against a 
mistake, then the truncate itself.

> On Feb 22, 2017, at 6:05 PM, Sanal Vasudevan  wrote:
> 
> Hi Folks,
> 
> I am trying to read Mutations from commit log files through an
> implementation of CommitLogReadHandler interface.
> 
> For a truncate CQL operation, I do not see a Mutation object.
> 
> Does C* skip writing the truncate operation into the commit log file?
> 
> 
> Thanks for your help.
> 
> Best regards,
> Sanal

Re: Rollback procedure for Cassandra Upgrade.

[Jeremy Hanna]

See the comment thread on https://issues.apache.org/jira/browse/CASSANDRA-8928 
 (add downgradesstables)
> On Jan 10, 2017, at 5:00 AM, Jonathan Haddad  wrote:
> 
> There's no downgrade procedure. You either upgrade or you go back to a
> snapshot from the previous version.
> On Mon, Jan 9, 2017 at 8:13 PM Prakash Chauhan 
> wrote:
> 
>> Hi All ,
>> 
>> Do we have an official procedure to rollback the upgrade of C* from 2.0.x
>> to 2.1.x ?
>> 
>> 
>> Description:
>> I have upgraded C* from 2.0.x to 2.1.x . As a part of upgrade procedure ,
>> I have to run nodetool upgradesstables .
>> What if the command fails in the middle ? Some of the sstables will be in
>> newer format (*-ka-*) where as other might be in older format(*-jb-*).
>> 
>> Do we have a standard procedure to do rollback in such cases?
>> 
>> 
>> 
>> Regards,
>> Prakash Chauhan.
>> 
>> 

Re: Broader community involvement in 4.0 (WAS Re: Rough roadmap for 4.0)

[Jeremy Hanna]

Regarding low hanging fruit, on the How To Contribute page [1] we’ve tried to 
keep a list of lhf tickets [2] linked to help people get started.  They are 
usually good starting points and don’t require much context.  I rarely see 
duplicates from lhf tickets.

Regarding duplicates, in my experience those who resolve tickets as duplicates 
are generally pretty good.

I think the safest bet to start is to look at How To Contribute page and the 
lhf labeled tickets.

[1] https://wiki.apache.org/cassandra/HowToContribute 

[2] 
https://issues.apache.org/jira/secure/IssueNavigator.jspa?reset=true=project+=+12310865+AND+labels+=+lhf+AND+status+!=+resolved
 


> On Nov 10, 2016, at 12:06 PM, Anuj Wadehra  
> wrote:
> 
> 
> Hi,
> 
> We need to understand that time is precious for all of us. Even if a 
> developer has intentions to contribute, he may take months to contribute his 
> first patch or may be longer. Some common entry barriers are:
> 1. Difficult to identify low hanging fruits. 30 JIRA comments on a ticket and 
> a new comer is LOST, even though the exact fix may be much simpler.
> 2. Dead JIRA discussions with no clue on the current status of the ticket.
> 3. No response on new JIRAs raised. Response time  to validate/reject the 
> problem is important. Should I pick? Is it really a bug? Maybe some expert 
> can confirm it first and then I can pick it..
> 4.Ping Pong JIRAs: Your read 10 comments of a ticket then see duplicates and 
> related ones..then read 30 more comments and then so on till you land up on 
> same JIRA which is not concluded yet.
> Possible Solution for above 4 points:
> A. Add a new JIRA field to crisply summarize what conclusive discussion has 
> taken place till now ,what's the status of current JIRA, proposed/feasible 
> solution etc.
> B. Mark low hanging fruits regularly.
> C. Validate/Reject newly reported JIRAs on priority. Using dev list to 
> validate/reject the issue before logging the JIRA??
> D. Make sure that duplicates are real proven duplicates.
> 
> 5. Insufficient code comments. 
> Solution: Coding comments should be a mandatory part of code review 
> checklist. It makes reviews faster and encourage people to understand the 
> flow and fix things on their own.
> 6. Insufficient Design documentation.
> Solution:Detailed documentation for at least new features so that people are 
> comfortable with the design. Reading English and understanding diagrams/flows 
> is much simpler than just jumping into the code upfront.
> 7. No/Little formal communication on active development and way forward.
> Solution: What about a monthly summary of New/Hot/critical JIRAs and new 
> feature development (with JIRA links so that topics of interest are 
> accessible)? 
> 
> ThanksAnuj
> 
> 
>  On Thu, 10 Nov, 2016 at 7:09 AM, Nate McCall wrote:   I 
> like the idea of a goal-based approach. I think that would make
> coming to a consensus a bit easier particularly if a larger number of
> people are involved.
> 
> On Tue, Nov 8, 2016 at 8:04 PM, Dikang Gu  wrote:
>> My 2 cents. I'm wondering is it a good idea to have some high level goals
>> for the major release? For example, the goals could be something like:
>> 1. Improve the scalability/reliability/performance by X%.
>> 2. Add Y new features (feature A, B, C, D...).
>> 3. Fix Z known issues  (issue A, B, C, D...).
>> 
>> I feel If we can have the high level goals, it would be easy to pick the
>> jiras to be included in the release.
>> 
>> Does it make sense?
>> 
>> Thanks
>> Dikang.
>> 
>> On Mon, Nov 7, 2016 at 11:22 AM, Oleksandr Petrov <
>> oleksandr.pet...@gmail.com> wrote:
>> 
>>> Recently there was another discussion on documentation and comments [1]
>>> 
>>> On one hand, documentation and comments will help newcomers to familiarise
>>> themselves with the codebase. On the other - one may get up to speed by
>>> reading the code and adding some docs. Such things may require less
>>> oversight and can play some role in improving diversity / increasing an
>>> amount of involved people.
>>> 
>>> Same thing with tests. There are some areas where tests need some
>>> refactoring / improvements, or even just splitting them from one file to
>>> multiple. It's a good way to experience the process and get involved into
>>> discussion.
>>> 
>>> For that, we could add some issues with subtasks (just a few for starters)
>>> or even just a wiki page with a doc/test wishlist where everyone could add
>>> a couple of points.
>>> 
>>> Docs and tests could be used in addition to lhf issues, helping people,
>>> having comprehensive and quick process and everything else that was
>>> mentioned in this thread.
>>> 
>>> Thank you.
>>> 
>>> [1]
>>> 

Re: DataStax role in Cassandra and the ASF

[Jeremy Hanna]

Ultimately it doesn't matter now. The project has a bright future with the 
involvement of all individuals regardless of the company they work for. That's 
the important thing.

> On Nov 5, 2016, at 10:30 AM, Jeremy Hanna <jeremy.hanna1...@gmail.com> wrote:
> 
> No it wasn't. You're citing the eventual and agreed upon outcome. I was 
> talking about the approach which is clear in the dev and user list threads 
> that the board was involved in. It is also apparently much more apparent in 
> the private threads which apparently the PMC can make public.
> 
>> On Nov 5, 2016, at 10:02 AM, Jim Jagielski <j...@jagunet.com> wrote:
>> 
>> Which is what was done: 
>> https://whimsy.apache.org/board/minutes/Cassandra.html
>> 
>>> On Nov 5, 2016, at 10:48 AM, Jeremy Hanna <jeremy.hanna1...@gmail.com> 
>>> wrote:
>>> 
>>> If the ASF is at risk with a single company allowed to dominate a project 
>>> then why couldn't the approach have been something like: "great job on 
>>> building a successful project and community. We think there is great 
>>> potential for more involvement at the core contribution level. How can we 
>>> work together to augment the existing efforts to encourage contribution and 
>>> bring in new contributors? By the way here are a couple of policy and 
>>> trademark things that we need to get fixed."
>>> 
>>> I didn't understand the assumption that DataStax was doing something 
>>> nefarious nor the approach that was taken.  On a personal note I had tried 
>>> to ask about evidence and the approach previously but was ignored: 
>>> https://www.mail-archive.com/dev@cassandra.apache.org/msg09101.html  
>>> Perhaps that was due to the volume of messages on that thread but I don't 
>>> feel those questions were ever addressed.
>>> 
>>> Regardless, I see a positive way forward for the project and am grateful to 
>>> everyone working towards that.
>>> 
>> 

Re: DataStax role in Cassandra and the ASF

[Jeremy Hanna]

No it wasn't. You're citing the eventual and agreed upon outcome. I was talking 
about the approach which is clear in the dev and user list threads that the 
board was involved in. It is also apparently much more apparent in the private 
threads which apparently the PMC can make public.

> On Nov 5, 2016, at 10:02 AM, Jim Jagielski <j...@jagunet.com> wrote:
> 
> Which is what was done: https://whimsy.apache.org/board/minutes/Cassandra.html
> 
>> On Nov 5, 2016, at 10:48 AM, Jeremy Hanna <jeremy.hanna1...@gmail.com> wrote:
>> 
>> If the ASF is at risk with a single company allowed to dominate a project 
>> then why couldn't the approach have been something like: "great job on 
>> building a successful project and community. We think there is great 
>> potential for more involvement at the core contribution level. How can we 
>> work together to augment the existing efforts to encourage contribution and 
>> bring in new contributors? By the way here are a couple of policy and 
>> trademark things that we need to get fixed."
>> 
>> I didn't understand the assumption that DataStax was doing something 
>> nefarious nor the approach that was taken.  On a personal note I had tried 
>> to ask about evidence and the approach previously but was ignored: 
>> https://www.mail-archive.com/dev@cassandra.apache.org/msg09101.html  Perhaps 
>> that was due to the volume of messages on that thread but I don't feel those 
>> questions were ever addressed.
>> 
>> Regardless, I see a positive way forward for the project and am grateful to 
>> everyone working towards that.
>> 
> 

Re: DataStax role in Cassandra and the ASF

[Jeremy Hanna]

If the ASF is at risk with a single company allowed to dominate a project then 
why couldn't the approach have been something like: "great job on building a 
successful project and community. We think there is great potential for more 
involvement at the core contribution level. How can we work together to augment 
the existing efforts to encourage contribution and bring in new contributors? 
By the way here are a couple of policy and trademark things that we need to get 
fixed."

I didn't understand the assumption that DataStax was doing something nefarious 
nor the approach that was taken.  On a personal note I had tried to ask about 
evidence and the approach previously but was ignored: 
https://www.mail-archive.com/dev@cassandra.apache.org/msg09101.html  Perhaps 
that was due to the volume of messages on that thread but I don't feel those 
questions were ever addressed.

Regardless, I see a positive way forward for the project and am grateful to 
everyone working towards that.

> On Nov 5, 2016, at 8:30 AM, Mark Struberg  wrote:
> 
> Having a bit insight how the board operates (being PMC-chair for 2 other 
> TLPs) I can ensure you that the board did handle this very cleanly!
> 
> A few things really should FIRST get handled in private. This is the same 
> regardless whether it's about board oversight or you as a PMC. 
> 
> An example is e.g. when we detect trademark violations. Or if ASF hosted 
> pages make unfair advertisement for ONE of the involved contributors. In such 
> cases the PMC (or board if the PMC doesn't act by itself) first tries to 
> solve those issues _without_ breaking porcelain! Which means the respective 
> person or company will get contacted in private and not immediately get hit 
> by public shaming and blaming. In most cases it's just an oversight and too 
> eager marketing people who don't understand the impact. Usually the problems 
> quickly get resolved without anyone loosing it's face.
> 
> 
> Oh, talking about the 'impact' and some people wondering why the ASF board is 
> so pissed?
> Well, the point is that in extremis the whole §501(c),3 (non-for-profit) 
> status is at risk! Means if we allow a single vendor to create an unfair 
> business benefit, then this might be interpreted as a profit making mechanism 
> by the federal tax office...
> This is one of the huge differences to some other OSS projects which are 
> basically owned by one company or where companies simply can buy a seat in 
> the board. 
> 
> 
> LieGrue,
> strub
> 
> PS: I strongly believe that the technical people at DataStax really tried to 
> do their best but got out-maneuvered by their marketing and sales people. The 
> current step was just part of a clean separation btw a company and their OSS 
> contributions. It was legally necessary and also important for the overall 
> Cassandra community!
> 
> 
> PPS: DataStax did a lot for Cassandra, but the public perception nowadays 
> seems to be that DataStax donated Cassandra to the ASF. This is not true. It 
> was created and contributed by Facebook 
> https://wiki.apache.org/incubator/Cassandramany years before DataStax was 
> even founded
> 
> 
> 
>> On Saturday, 5 November 2016, 13:12, Benedict Elliott Smith 
>>  wrote:
>> 
>> I would hope the board would engage with criticism substantively, and that 
>> "long emails" to boards@ would be responded to on their merit, without a 
>> grassroots effort to apply pressure.
>> 
>> 
>> In lieu of that, it is very hard for the community to "speak with one voice" 
>> because we do not know what actions the board has undertaken.  This is at 
>> odds with "The Apache Way" core tenet of Openness.
>> 
>> 
>> The actions I have seen on the public fora by both Chris and Mark make me 
>> doubt the actions in private were reasonable.
>> 
>> 
>> 
>> I reiterate that the board should make all of its discussions about 
>> DataStax, particularly those with the PMC-private list, public.  Otherwise 
>> the community cannot perform the function you ask.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> On 5 November 2016 at 03:08, Ross Gardler  wrote:
>> 
>> [In the mail below I try not to cast judgement, I do not know enough of the 
>> background to have an opinion on this specific situation. My comments are in 
>> response to the question “Where are the board's guidelines then, or do they 
>> make it up as they go?”.]
>>> 
>>> The boards guidelines are the Apache Way. This is a fluid thing that adapts 
>>> to individual project needs but has a few common pillars in all projects, 
>>> e.g. PMC is responsible for community health and PMC members are expected 
>>> to act as individuals in the interest of the community. The board is 
>>> empowered, by the ASF membership (individuals with merit) to take any 
>>> action necessary to ensure a PMC is carrying out its duty.
>>> 
>>> If a PMC is being ineffective then the board only has blunt instruments to 
>>> 

Re: DataStax role in Cassandra and the ASF

[Jeremy Hanna]

"I’m sorry you had a bad experience, but it feels like that’s the normal ebb 
and flow of projects.”

I’m sorry - that should have been - I’m sorry some ideas were not accepted, but 
it feels like that’s the normal ebb and flow of projects.  I am sorry you had a 
bad experience - without any qualification :)

Kind regards,

Jeremy

> On Nov 4, 2016, at 10:58 PM, Jeremy Hanna <jeremy.hanna1...@gmail.com> wrote:
> 
> Hi Łukasz,
> 
> I’m sorry you found the projects difficult to work with.  It sounds like in 
> this case it was about modularizing with Maven and making TinkerPop work 
> better with OSGI.  People in the project have been going back and forth about 
> the build process since before Riptano and DataStax existed and the decisions 
> by the PMC and community remained constant - they just wanted to stick with a 
> more explicit build system with ant - sure it’s preference based, but that’s 
> where things were before DataStax was even started.  With regard to OSGI, it 
> sounds like it was just not an item that they saw as a priority at the time 
> but were open to considering in the future.  I thought Stephen was very open 
> and generally he bends over backwards to help people as you can find in many 
> other interactions on Stack Overflow, various gremlin, titan, and tinkerpop 
> mailing lists.  I’ve opened a lot of TinkerPop tickets, some are accepted, 
> some aren’t.  I need to do better about doing my part as well to do things 
> with tickets that I commit to in TinkerPop.  I’m sorry you had a bad 
> experience, but it feels like that’s the normal ebb and flow of projects.  
> That said, we can do a better job in either explaining or being more 
> welcoming - that’s nothing to do with any company though.  That’s a community 
> thing.
> 
> Kind regards,
> 
> Jeremy
> 
>> On Nov 4, 2016, at 8:03 PM, Łukasz Dywicki <l...@code-house.org> wrote:
>> 
>> Good evening,
>> I feel myself a bit called to table by both Kelly and Chris. Thing is I 
>> don’t know personally nor have any relationship with both of you. I’m not 
>> even ASF member. My tweet was simply reaction for Kelly complaints about ASF 
>> punishing out DataStax. Kelly timeline also contained statement such 
>> "forming a long term strategy to grow diversity around” which reminded me my 
>> attempts to collaborate on Cassandra and Tinkerpop projects to grow such 
>> diversity. I collected message links and quotes and put it into gist who 
>> could be read by anyone: 
>> https://gist.github.com/splatch/aebe4ad4d127922642bee0dc9a8b1ec1 
>> 
>> I don’t want to bring now these topics back and disscuss technical stuff 
>> over again. It happened to me in the past to refuse (or vote against) some 
>> change proposals in other Apache projects I am involved. I was on the other 
>> ("bad guy") side multiple times. I simply collected public records of 
>> interactions with DataStax staff I was aware, simply because of my personal 
>> involvement. It shown how some ideas, yet cassandra mailing list don’t have 
>> many of these coming from externals, are getting put a side with very little 
>> or even lack of will to pull in others people work in. This is blocking 
>> point for anyone coming from external sides to get involved into project and 
>> help it growing. If someone changes requires moves in project core or it’s 
>> public APIs that person will require support from project members to get 
>> this done. If such help will not be given it any outside change will be ever 
>> completed and noone will invest time in doing something more than fixing 
>> typos or common programmer errors which we all do from time to time. Despite 
>> of impersonal nature of communications in Internet we still do have human 
>> interactions and we all have just one chance to make first impression. If we 
>> made it wrong at beginning its hard to fix it later on. 
>> Some decisions made in past by project PMCs lead to situation that project 
>> was forked and maintained outside ASF (ie. stratio cassandra which 
>> eventually ended up as lucene indexes plugin over a year ago), some other 
>> did hurt users running cassandra for long time (ie. discontinuation of 
>> thrift). Especially second decission was seen by outsiders, who do not 
>> desire billion writes per second, as marketing driven. This led to people 
>> looking and finding alternatives using compatible interface which might be, 
>> ironically, even faster (ie. scylladb).
>> 
>> And since there was quote battle on twitter between Jim Jagielski and 
>> Benedict, I can throw some in as well. Over conferences I attended and even 
>> during consu

Re: DataStax role in Cassandra and the ASF

[Jeremy Hanna]

Hi Łukasz,

I’m sorry you found the projects difficult to work with.  It sounds like in 
this case it was about modularizing with Maven and making TinkerPop work better 
with OSGI.  People in the project have been going back and forth about the 
build process since before Riptano and DataStax existed and the decisions by 
the PMC and community remained constant - they just wanted to stick with a more 
explicit build system with ant - sure it’s preference based, but that’s where 
things were before DataStax was even started.  With regard to OSGI, it sounds 
like it was just not an item that they saw as a priority at the time but were 
open to considering in the future.  I thought Stephen was very open and 
generally he bends over backwards to help people as you can find in many other 
interactions on Stack Overflow, various gremlin, titan, and tinkerpop mailing 
lists.  I’ve opened a lot of TinkerPop tickets, some are accepted, some aren’t. 
 I need to do better about doing my part as well to do things with tickets that 
I commit to in TinkerPop.  I’m sorry you had a bad experience, but it feels 
like that’s the normal ebb and flow of projects.  That said, we can do a better 
job in either explaining or being more welcoming - that’s nothing to do with 
any company though.  That’s a community thing.

Kind regards,

Jeremy

> On Nov 4, 2016, at 8:03 PM, Łukasz Dywicki  wrote:
> 
> Good evening,
> I feel myself a bit called to table by both Kelly and Chris. Thing is I don’t 
> know personally nor have any relationship with both of you. I’m not even ASF 
> member. My tweet was simply reaction for Kelly complaints about ASF punishing 
> out DataStax. Kelly timeline also contained statement such "forming a long 
> term strategy to grow diversity around” which reminded me my attempts to 
> collaborate on Cassandra and Tinkerpop projects to grow such diversity. I 
> collected message links and quotes and put it into gist who could be read by 
> anyone: 
> https://gist.github.com/splatch/aebe4ad4d127922642bee0dc9a8b1ec1 
> 
> I don’t want to bring now these topics back and disscuss technical stuff over 
> again. It happened to me in the past to refuse (or vote against) some change 
> proposals in other Apache projects I am involved. I was on the other ("bad 
> guy") side multiple times. I simply collected public records of interactions 
> with DataStax staff I was aware, simply because of my personal involvement. 
> It shown how some ideas, yet cassandra mailing list don’t have many of these 
> coming from externals, are getting put a side with very little or even lack 
> of will to pull in others people work in. This is blocking point for anyone 
> coming from external sides to get involved into project and help it growing. 
> If someone changes requires moves in project core or it’s public APIs that 
> person will require support from project members to get this done. If such 
> help will not be given it any outside change will be ever completed and noone 
> will invest time in doing something more than fixing typos or common 
> programmer errors which we all do from time to time. Despite of impersonal 
> nature of communications in Internet we still do have human interactions and 
> we all have just one chance to make first impression. If we made it wrong at 
> beginning its hard to fix it later on. 
> Some decisions made in past by project PMCs lead to situation that project 
> was forked and maintained outside ASF (ie. stratio cassandra which eventually 
> ended up as lucene indexes plugin over a year ago), some other did hurt users 
> running cassandra for long time (ie. discontinuation of thrift). Especially 
> second decission was seen by outsiders, who do not desire billion writes per 
> second, as marketing driven. This led to people looking and finding 
> alternatives using compatible interface which might be, ironically, even 
> faster (ie. scylladb).
> 
> And since there was quote battle on twitter between Jim Jagielski and 
> Benedict, I can throw some in as well. Over conferences I attended and even 
> during consultancy services I got, I’ve spoken with some people having 
> records of DataStax in their resumes and even them told me "collaboration 
> with them [cassandra team] was hard". Now imagine how outsider will get any 
> chance to get any change done with such attitude shown even to own 
> colleagues? Must also note that Tinkerpop is getting better on this field 
> since it has much more generic nature.
> I don’t think this whole topic is to say that you (meaning DataStax) made 
> wrong job, or you are doing wrong for project but about letting others join 
> forces with you to make Cassandra even better. Maybe there is not a lot of 
> people currently walking around but once you will welcome and help them 
> working with you on code base you may be sure that others will join making 
> your development efforts easier and shared across community. 
> 
> Kind regards,
> Lukasz

Re: Low hanging fruit crew

[Jeremy Hanna]

It sounds like everyone is on the same page - there won’t be a rubber stamp.  
It’s just to have a concerted effort to help these tickets move along as 
they’re often the first experience that contributors have with the project.  
I’m sure many of you know of the ‘lhf' tag that’s out there with an associated 
Jira search [1] on the How to Contribute page [2].  In the Jira search, you can 
see several that are either “patch available” or “awaiting feedback” so that 
search a place to start.  As for something fancier like a dashboard, I 
currently can’t make a share-able dashboard as a contributor to the project.  
Perhaps committers have more permissions there.

1) 
https://issues.apache.org/jira/issues/?jql=project%20%3D%2012310865%20AND%20labels%20%3D%20lhf%20AND%20status%20!%3D%20resolved
2) https://wiki.apache.org/cassandra/HowToContribute



> On Oct 19, 2016, at 5:27 PM, Jonathan Haddad <j...@jonhaddad.com> wrote:
> 
> Tagging tickets as LHF is a great idea.  There's plenty of people that
> would love to set up a JIRA dashboard saved search / nightly email for LHF
> tickets.
> 
> On Wed, Oct 19, 2016 at 1:34 PM Jeff Beck <beckj...@gmail.com> wrote:
> 
>> Would it make sense to allow people submitting the patch to flag things as
>> LHF or small tasks? If it doesn't look like it is simple enough the team
>> could remove the label but it may help get feedback to patches more
>> quickly, even something saying accepted for review would be nice.
>> 
>> Personally if a ticket sits for a few weeks I have no idea what next steps
>> I should take to keep it moving forward. We may want to document what a
>> person submitting a patch should do next and if it is documented we should
>> add a link on
>> http://cassandra.apache.org/doc/latest/development/patches.html
>> 
>> Jeff
>> 
>> 
>> PS: My example is https://issues.apache.org/jira/browse/CASSANDRA-12633
>> 
>> On Wed, Oct 19, 2016 at 12:21 PM Edward Capriolo <edlinuxg...@gmail.com>
>> wrote:
>> 
>>> Also no one has said anything to the effect of 'we want to rubber stamp
>>> reviews' so that ...evil reason. Many of us are coders by trade and
>>> understand why that is bad.
>>> 
>>> On Wednesday, October 19, 2016, Edward Capriolo <edlinuxg...@gmail.com>
>>> wrote:
>>> 
>>>> I realize that test passing a small tests and trivial reviews will not
>>>> catch all issues. I am  not attempting to trivialize the review
>> process.
>>>> 
>>>> Both deep and shallow bugs exist. The deep bugs, I am not convinced
>> that
>>>> even an expert looking at the contribution for N days can account for a
>>>> majority of them.
>>>> 
>>>> The shallow ones may appear shallow and may be deep, but given that a
>> bug
>>>> already exists an attempt to fix it usually does not arrive at
>> something
>>>> worse.
>>>> 
>>>> Many of these issues boil down to simple, seeemingly clear fixes. Some
>>> are
>>>> just basic metric addition. Many have no interaction for weeks.
>>>> 
>>>> 
>>>> On Wednesday, October 19, 2016, Jeff Jirsa <jeff.ji...@crowdstrike.com
>>>> <javascript:_e(%7B%7D,'cvml','jeff.ji...@crowdstrike.com');>> wrote:
>>>> 
>>>>> Let’s not get too far in the theoretical weeds. The email thread
>> really
>>>>> focused on low hanging tickets – tickets that need review, but
>>> definitely
>>>>> not 8099 level reviews:
>>>>> 
>>>>> 1) There’s a lot of low hanging tickets that would benefit from
>> outside
>>>>> contributors as their first-patch in Cassandra (like
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-12541 ,
>>>>> https://issues.apache.org/jira/browse/CASSANDRA-12776 )
>>>>> 2) Some of these patches already exist and just need to be reviewed
>> and
>>>>> eventually committed.
>>>>> 
>>>>> Folks like Ed and Kurt have been really active in Jira lately, and
>> there
>>>>> aren’t a ton of people currently reviewing/committing – that’s part of
>>> OSS
>>>>> life, but the less friction that exists getting those patches reviewed
>>> and
>>>>> committed, the more people will be willing to contribute in the
>> future,
>>> and
>>>>> the better off the project will be.
>>>>> 
>>>>> 
>>>>> On 10/19/16, 9:14 AM, "Jeremy Hanna" <jeremy.hanna1...@g

Re: Low hanging fruit crew

[Jeremy Hanna]

And just to be clear, I think everyone would welcome more testing for both 
regressions of new code correctness.  I think everyone would appreciate the 
time savings around more automation.  That should give more time for a 
thoughtful review - which is likely what new contributors really need to get 
familiar with different parts of the codebase.  These LHF reviews won’t be like 
the code reviews of the vnode or the 8099 ticket obviously, so it won’t be a 
huge burden.  But it has some very tangible benefits imo, as has been said.

> On Oct 19, 2016, at 11:08 AM, Jonathan Ellis  wrote:
> 
> I specifically used the phrase "problems that the test would not" to show I
> am talking about more than mechanical correctness.  Even if the tests are
> perfect (and as Jeremiah points out, how will you know that without reading
> the code?), you can still pass tests with bad code.  And is expecting
> perfect tests really realistic for multithreaded code?
> 
> But besides correctness, reviews should deal with
> 
> 1. Efficiency.  Is there a quadratic loop that will blow up when the number
> of nodes in a cluster gets large?  Is there a linear amount of memory used
> that will cause problems when a partition gets large?  These are not
> theoretical problems.
> 
> 2. Maintainability.  Is this the simplest way to accomplish your goals?  Is
> there a method in SSTableReader that would make your life easier if you
> refactored it a bit instead of reinventing it?  Does this reduce technical
> debt or add to it?
> 
> 3. "Bus factor."  If only the author understands the code and all anyone
> else knows is that tests pass, the project will quickly be in bad shape.
> Review should ensure that at least one other person understand the code,
> what it does, and why, at a level that s/he could fix bugs later in it if
> necessary.
> 
> On Wed, Oct 19, 2016 at 10:52 AM, Jonathan Haddad  wrote:
> 
>> Shouldn't the tests test the code for correctness?
>> 
>> On Wed, Oct 19, 2016 at 8:34 AM Jonathan Ellis  wrote:
>> 
>>> On Wed, Oct 19, 2016 at 8:27 AM, Benjamin Lerer <
>>> benjamin.le...@datastax.com
 wrote:
>>> 
 Having the test passing does not mean that a patch is fine. Which is
>> why
>>> we
 have a review check list.
 I never put a patch available without having the tests passing but most
>>> of
 my patches never pass on the first try. We always make mistakes no
>> matter
 how hard we try.
 The reviewer job is to catch those mistakes by looking at the patch
>> from
 another angle. Of course, sometime, both of them fail.
 
>>> 
>>> Agreed.  Review should not just be a "tests pass, +1" rubber stamp, but
>>> actually checking the code for correctness.  The former is just process;
>>> the latter actually catches problems that the tests would not.  (And this
>>> is true even if the tests are much much better than ours.)
>>> 
>>> --
>>> Jonathan Ellis
>>> co-founder, http://www.datastax.com
>>> @spyced
>>> 
>> 
> 
> 
> 
> -- 
> Jonathan Ellis
> co-founder, http://www.datastax.com
> @spyced

Re: Proposal - 3.5.1

[Jeremy Hanna]

Right - I think like Jake and others have said, it seems appropriate to do 
something at this point.  Would a clearer, more liberal backport policy to the 
odd versions be worthwhile until we find our footing?  As Jeremiah said, it 
does seem like the big bang 3.0 release has caused much of the baggage that 
we’re facing.  Combine with that the slow uptake on any specific version so far 
at least partly because of the newness of the release model.

To me, the hard thing to me about 3 month releases is that then you get into 
the larger untested feature releases which is what it was originally supposed 
to get away from.

So in essence, would we
1) do nothing and see it through
2) have a more liberal backport policy in the 3.x line and revisit once we get 
to 4
3) do a tick-tock(-tock-tock) sort of model
4) do some sort of LTS
5) go back to the drawing board
6) go back to the old model

I think the earlier numbers imply some confidence in the thinking behind 
tick-tock.  Would 2 be acceptable to see the 3.x line through with the current 
release model?  Or do we need to do something more extensive at this stage?

> On Sep 15, 2016, at 1:59 PM, Jonathan Haddad  wrote:
> 
> I don't think it's binary - we don't have to do year long insanity or
> bleeding edge crazyness.
> 
> How about a release every 3 months, with each release accepting 6 months of
> patches?  (oldstable & newstable)  Also provide nightly builds & stick to
> the idea of stable trunk.
> 
> The issue is the number of bug fixes a given release gets.  1 bug fix
> release for a new feature is just terrible.  The community as a whole
> despises this system and is lowering confidence in the project.
> 
> Jon
> 
> 
> On Thu, Sep 15, 2016 at 11:48 AM Jake Luciani  wrote:
> 
>> I'm pretty sure everyone will agree Tick-Tock didn't go well and needs to
>> change.
>> 
>> The problem for me is going back to the old way doesn't sound great. There
>> are parts of tick-tock I really like,
>> for example, the cadence and limited scope per release.
>> 
>> I know at the summit there were a lot of ideas thrown around I can
>> regurgitate but perhaps people
>> who have been thinking about this would like to chime in and present ideas?
>> 
>> -Jake
>> 
>> On Thu, Sep 15, 2016 at 2:28 PM, Benedict Elliott Smith <
>> bened...@apache.org
>>> wrote:
>> 
>>> I agree tick-tock is a failure.  But for two reasons IMO:
>>> 
>>> 1) Ultimately, the users are the real testers and it takes a while for a
>>> release to percolate into the wild for feedback.  The reality is that a
>>> release doesn't have its tires properly kicked for at least three months
>>> after it's cut.  So if we are to have any tocks, they should be
>> completely
>>> unwed from the ticks, and should probably happen on a ~3M cadence to keep
>>> the labour down but the utility up (and there should probably still be
>> more
>>> than one tock per tick)
>>> 
>>> 2) Those promised resources to improved process never happened.  We
>> haven't
>>> even reached parity with the 2.1 release until very recently, i.e. no
>>> failing u/dtests.
>>> 
>>> 
>>> On 15 September 2016 at 19:08, Jeff Jirsa 
>>> wrote:
>>> 
 I know we’ve got a lot of folks following the dev list without a lot of
 background, so let’s make sure we get some context here so everyone can
>>> be
 on the same page.
 
 Going to preface this wall of text by saying I’m +1 on a 3.5.1 (and
>>> 3.3.1,
 etc) if it’s done AFTER 3.9 (I think we need to get 3.9 out first
>> before
 the RE manpower is spent on backporting fixes, even critical fixes,
>>> because
 3.9 has multiple critical fixes for people running 3.7).
 
 Now some background:
 
 For many years, Cassandra used to have a dev process that kept 3 active
 branches - “bleeding edge”, a “stable”, and an “old stable” branch,
>> where
 developers would be committing ALL new contributions to the bleeding
>>> edge,
 non-api-breaking changes to stable, and bugfixes only to old stable.
>>> While
 the api changed and major features were added, that bleeding edge would
 just be ‘trunk’, and it’d get cut into a major version when it was
>> ready
>>> to
 ship. We saw that with 2.2 / 2.1 / 2.0 (and before that, 2.1 / 2.0 /
>> 1.2,
 and before that 2.0 / 1.2 / 1.1 ). When that bleeding edge got released
>>> as
 a major x.y.0, the third, oldest, most stable branch went EOL, and new
 features would go into trunk for the next major version.
 
 There were two big negatives observed with this:
 
 The first big negative is that if multiple major new features were in
 flight, releases were prone to delay. Nobody wants to break an API on a
 x.y.1 release, and nobody wants to add a new feature to a x.y.2
>> release,
>>> so
 the project would delay the x.y releases if major features were close,
>>> and
 then there’d be pressure to slip them in 

Re: Proposal: create a mailing list for just the newly created Jira ticket notifications

[Jeremy Hanna]

It sounds like people generally prefer subscribing to Jira filters.  I’ll just 
keep my message rules then from the commits@ mailing list.

Thanks all.

Jeremy

> On Aug 29, 2016, at 12:32 PM, Patrick McFadin <pmcfa...@gmail.com> wrote:
> 
> I have one setup for new AND updated. Very handy to seeing activity
> progression.
> 
> Jira has some really nice query + notification settings if you want to get
> more granular.
> 
> On Mon, Aug 29, 2016 at 10:12 AM, Jonathan Haddad <j...@jonhaddad.com> wrote:
> 
>> Wow, I am embarrassed to say that I had no idea that existed.  Thanks Russ!
>> 
>> On Mon, Aug 29, 2016 at 9:58 AM Russell Hatch <rha...@datastax.com> wrote:
>> 
>>> This is currently possible using jira saved search + subscription.
>> Create a
>>> new saved search with a query like:
>>> 
>>> project = CASSANDRA AND created >= -24h ORDER BY createdDate DESC
>>> 
>>> When viewing your filter, go to Details > New Subscription and set it up
>> to
>>> run once a day and you're set!
>>> 
>>> Hope it helps,
>>> 
>>> Russ
>>> 
>>> On Mon, Aug 29, 2016 at 10:45 AM, Jonathan Haddad <j...@jonhaddad.com>
>>> wrote:
>>> 
>>>> A daily digest of new issues would be incredible.  A non-binding +1
>> from
>>>> this guy.
>>>> 
>>>> 
>>>> On Mon, Aug 29, 2016 at 8:58 AM Jeremy Hanna <
>> jeremy.hanna1...@gmail.com
>>>> 
>>>> wrote:
>>>> 
>>>>> Currently we have a mailing list (commits@) that includes commit
>>>> messages
>>>>> as well as all updates to Jira tickets.  However for the purpose of
>>>> staying
>>>>> up to date with new tickets, it’s something of a firehose with the
>>> rapid
>>>>> rate of change.
>>>>> 
>>>>> Some people (like me) filter out just the new ticket creation emails
>>> and
>>>>> watch/vote for those that seem interesting.  Others create Jira
>> filters
>>>> and
>>>>> subscribe to them.  Personally I prefer an email per Jira creation.
>>> For
>>>>> those that prefer that route, would anyone mind if I created an infra
>>>>> ticket to create a mailing list to just publish those emails?  Are
>>> there
>>>>> others like me out there that prefer that to doing a daily digest or
>>>>> something similar?
>>>>> 
>>>>> Thanks,
>>>>> 
>>>>> Jeremy
>>>> 
>>> 
>> 

Re: Support Multi-Tenant in Cassandra

[Jeremy Hanna]

For posterity, our wiki page from many moons ago was 
https://wiki.apache.org/cassandra/MultiTenant 
<https://wiki.apache.org/cassandra/MultiTenant>.  It was a different era of the 
project but there might be some useful bits in there for anyone interested in 
MT.

> On Sep 9, 2016, at 9:28 PM, Jason Brown <jasedbr...@gmail.com> wrote:
> 
> The current implementation will probably be yanked when thrift as a whole
> is removed for 4.0. And I'm ok with that.
> 
> That being said, there has been an undercurrent of interest over time about
> multitenancy, and I'm willing to entertain a renewed discussion. It might
> be instructive to see if any other systems are currently offering
> multitenancy and if there's something to be learned there. If not, we could
> at least explore the topic more seriously and then document for posterity
> the well-informed pros/cons of why we as a community choose to not do it,
> postpone for later, or actually do it. Of course, it would be great for a
> motivated individual to lead the effort if we really want to entertain it.
> 
> On Friday, September 9, 2016, Jeremy Hanna <jeremy.hanna1...@gmail.com>
> wrote:
> 
>> I agree that the request scheduler should probably be deprecated and
>> removed unless someone wants to put in something that's usable from the non
>> thrift request processor. We added it for prioritization and QoS but I
>> don't know of anyone ever using it. Our project we thought of using it for
>> got shelved.
>> 
>> Unless it's just multiple clients with the same general use case, I think
>> multi tenant is going to be quite difficult to tune and diagnose problems
>> for. I would steer clear and have a cluster per logical app if at all
>> possible.
>> 
>>> On Sep 9, 2016, at 6:43 PM, Mick Semb Wever <m...@thelastpickle.com
>> <javascript:;>> wrote:
>>> 
>>> On 15 July 2016 at 16:38, jason zhao yang <zhaoyangsingap...@gmail.com
>> <javascript:;>>
>>> wrote:
>>> 
>>>> 
>>>> May I ask is there any plan of extending functionalities related to
>>>> Multi-Tenant?
>>> 
>>> 
>>> 
>>> I had needs for this in the past and my questioning always seemed to
>>> eventuate to answers along the lines of this should be done more at the
>>> resource level. There is a variety of ways a bad datamodel or client can
>>> bring a cluster down, not just at request time.
>>> 
>>> There was some thoughts IIRC around a resource scheduler somewhere
>> post-3.0
>>> but i don't think that ever eventuated (someone more knowledgable please
>>> correct me).
>>> 
>>> Otherwise you could look into using tiered storage so that you had at
>> least
>>> disk isolation per keyspace. Solves some things, but won't help with
>>> overhead and memtable impact from number of keyspaces/tables and lack of
>>> heap/throughput isolation/scheduling.
>>> 
>>> The approach of doing this at the driver level, prefixing the partition
>>> key, is as good as any approach for now.
>>> 
>>> Could be an idea to remove/deprecate the request_scheduler from code and
>>> yaml.
>>> 
>>> ~mck
>> 

Re: Support Multi-Tenant in Cassandra

[Jeremy Hanna]

I agree that the request scheduler should probably be deprecated and removed 
unless someone wants to put in something that's usable from the non thrift 
request processor. We added it for prioritization and QoS but I don't know of 
anyone ever using it. Our project we thought of using it for got shelved.

Unless it's just multiple clients with the same general use case, I think multi 
tenant is going to be quite difficult to tune and diagnose problems for. I 
would steer clear and have a cluster per logical app if at all possible.

> On Sep 9, 2016, at 6:43 PM, Mick Semb Wever  wrote:
> 
> On 15 July 2016 at 16:38, jason zhao yang 
> wrote:
> 
>> 
>> May I ask is there any plan of extending functionalities related to
>> Multi-Tenant?
> 
> 
> 
> I had needs for this in the past and my questioning always seemed to
> eventuate to answers along the lines of this should be done more at the
> resource level. There is a variety of ways a bad datamodel or client can
> bring a cluster down, not just at request time.
> 
> There was some thoughts IIRC around a resource scheduler somewhere post-3.0
> but i don't think that ever eventuated (someone more knowledgable please
> correct me).
> 
> Otherwise you could look into using tiered storage so that you had at least
> disk isolation per keyspace. Solves some things, but won't help with
> overhead and memtable impact from number of keyspaces/tables and lack of
> heap/throughput isolation/scheduling.
> 
> The approach of doing this at the driver level, prefixing the partition
> key, is as good as any approach for now.
> 
> Could be an idea to remove/deprecate the request_scheduler from code and
> yaml.
> 
> ~mck

Proposal: create a mailing list for just the newly created Jira ticket notifications

[Jeremy Hanna]

Currently we have a mailing list (commits@) that includes commit messages as 
well as all updates to Jira tickets.  However for the purpose of staying up to 
date with new tickets, it’s something of a firehose with the rapid rate of 
change.

Some people (like me) filter out just the new ticket creation emails and 
watch/vote for those that seem interesting.  Others create Jira filters and 
subscribe to them.  Personally I prefer an email per Jira creation.  For those 
that prefer that route, would anyone mind if I created an infra ticket to 
create a mailing list to just publish those emails?  Are there others like me 
out there that prefer that to doing a daily digest or something similar?

Thanks,

Jeremy

Re: A proposal to move away from Jira-centric development

[Jeremy Hanna]

I think a separate mailing list for just ticket creation would be nice as well. 
 I think that’s what many of us filter down the commits@ list to.  That doesn’t 
have to happen in place of the proposed change but would make it easier for 
people to follow new issue creation.  From there I go to and follow/comment 
on/etc. issues I’m interested in.

> On Aug 16, 2016, at 4:06 PM, Eric Evans  wrote:
> 
> On Mon, Aug 15, 2016 at 9:22 AM, Jonathan Ellis  wrote:
> 
> [ ... ]
> 
>> I propose that we take advantage of the dev list to perform that
>> separation.  Major new features and architectural improvements should be
>> discussed first here, then when consensus on design is achieved, moved to
>> Jira for implementation and review.
>> 
>> I think this will also help with the problem when the initial idea proves
>> to be unworkable and gets revised substantially later after much
>> discussion.  It can be difficult to figure out what the conclusion was, as
>> review comments start to pile up afterwards.  Having that discussion on the
>> list, and summarizing on Jira, would mitigate this.
> 
> TL;DR +1
> 
> I think there are actually a couple of related, but disjoint issues here.
> 
> IMO, a JIRA should be the source of truth for an issue, a way to track
> any on-going efforts, and a historical account after-the-fact.
> Regardless of where you think discussions should take place, I would
> argue there is room for improvement here; Many of our JIRAs (I would
> argue the most interesting ones!), are very difficult to make use of
> for either of these cases (current status, or after-the-fact).  Some
> curation (as someone else pointed out in this thread), could go a long
> way.  Retitling and/or revising the description as the scope of a
> ticket evolves, or posting a summary or current status in the
> description body would be ways for people who are up to speed on an
> issue, to spend a few minutes making it valuable to others.  So would
> summarizing discussions that take place elsewhere.
> 
> The other issue is discoverability and/or inclusivity.  If the only
> way to keep abreast of what is happening is to follow the fire-hose of
> all JIRA updates, then contribution is going to be limited to those
> with the bandwidth.  If you work on Cassandra full-time, this probably
> doesn't seem like a big deal, but if your time is limited, then it can
> create quite a barrier (and I've been on both sides of this with
> Cassandra).  So moving serious discussions to the mailing list is also
> a sort of curation, since it creates a venue free of all the
> minutiae/noise.
> 
> My personal opinion is also that it's far easier to manage a given
> volume with email, and that the discussions are easier to follow (the
> interface is better at representing the ontology, for example), but
> from what I can gather, not everyone agrees so YMMV.
> 
> Cheers,
> 
> -- 
> Eric Evans
> john.eric.ev...@gmail.com

Re: A proposal to move away from Jira-centric development

[Jeremy Hanna]

I’m not a committer or PMC member.  I’m a dev list follower and contributor.  
I’ve been working with different apache projects for years.  I often don’t 
follow or filter the asf lists because I’m only interested in individual 
tickets.  I often don’t care how the decision was made, though that may be 
important for auditing purposes for a project.  I care that it’s been 
implemented and having an easy way to link to it if I want to give others an 
easy way to watch or vote for the feature.  Also I’ve found the lists as a pain 
because if I want to contribute something to a discussion I have to join the 
list.  I often don’t want to join a list about project X.  I just care insofar 
as it relates to what I want.  So I have my universal Jira account and I can 
watch or vote for or comment on tickets.  Within the Apache ecosystem, that’s 
much simpler than having to follow a list per project.

> On Aug 15, 2016, at 1:27 PM, Chris Mattmann <mattm...@apache.org> wrote:
> 
> s/dev list followers//
> 
> That’s (one of) the disconnect(s). It’s not *you the emboldened, powerful 
> PMC* 
> and then everyone else.
> 
> 
> On 8/15/16, 11:25 AM, "Jeremy Hanna" <jeremy.hanna1...@gmail.com> wrote:
> 
>Regarding high level linking, if I’m in irc or slack or hipchat or a 
> mailing list thread, it’s easy to reference a Jira ID and chat programs can 
> link to it and bots can bring up various details.  I don’t think a hash id 
> for a mailing list is as simple or memorable.
> 
>A feature of a mailing list thread is that it can go in different 
> directions easily.  The burden is that it will be harder to follow in the 
> future if you’re trying to sort out implementation details.  So for high 
> level discussion, the mailing list is great.  When getting down to the actual 
> work and discussion about that focused work, that’s where a tool like Jira 
> comes in.  Then it is reference-able in the changes.txt and other things.
> 
>I think the approach proposed by Jonathan is a nice way to keep dev list 
> followers informed but keeping ticket details focused.
> 
>> On Aug 15, 2016, at 1:12 PM, Chris Mattmann <mattm...@apache.org> wrote:
>> 
>> How is it harder to point someone to mail?
>> 
>> Have you seen lists.apache.org?
>> 
>> Specifically:
>> https://lists.apache.org/list.html?dev@cassandra.apache.org
>> 
>> 
>> 
>> On 8/15/16, 10:08 AM, "Jeremiah D Jordan" <jeremiah.jor...@gmail.com> wrote:
>> 
>>   I like keeping things in JIRA because then everything is in one place, and 
>> it is easy to refer someone to it in the future.
>>   But I agree that JIRA tickets with a bunch of design discussion and POC’s 
>> and such in them can get pretty long and convoluted.
>> 
>>   I don’t really like the idea of moving all of that discussion to email 
>> which makes it has harder to point someone to it.  Maybe a better idea would 
>> be to have a “design/POC” JIRA and an “implementation” JIRA.  That way we 
>> could still keep things in JIRA, but the final decision would be kept 
>> “clean”.
>> 
>>   Though it would be nice if people would send an email to the dev list when 
>> proposing “design” JIRA’s, as not everyone has time to follow every JIRA 
>> ever made to see that a new design JIRA was created that they might be 
>> interested in participating on.
>> 
>>   My 2c.
>> 
>>   -Jeremiah
>> 
>> 
>>> On Aug 15, 2016, at 9:22 AM, Jonathan Ellis <jbel...@gmail.com> wrote:
>>> 
>>> A long time ago, I was a proponent of keeping most development discussions
>>> on Jira, where tickets can be self contained and the threadless nature
>>> helps keep discussions from getting sidetracked.
>>> 
>>> But Cassandra was a lot smaller then, and as we've grown it has become
>>> necessary to separate out the signal (discussions of new features and major
>>> changes) from the noise of routine bug reports.
>>> 
>>> I propose that we take advantage of the dev list to perform that
>>> separation.  Major new features and architectural improvements should be
>>> discussed first here, then when consensus on design is achieved, moved to
>>> Jira for implementation and review.
>>> 
>>> I think this will also help with the problem when the initial idea proves
>>> to be unworkable and gets revised substantially later after much
>>> discussion.  It can be difficult to figure out what the conclusion was, as
>>> review comments start to pile up afterwards.  Having that discussion on the
>>> list, and summarizing on Jira, would mitigate this.
>>> 
>>> -- 
>>> Jonathan Ellis
>>> Project Chair, Apache Cassandra
>>> co-founder, http://www.datastax.com
>>> @spyced
>> 
>> 
>> 
>> 
> 
> 
> 
> 

Re: A proposal to move away from Jira-centric development

[Jeremy Hanna]

Regarding high level linking, if I’m in irc or slack or hipchat or a mailing 
list thread, it’s easy to reference a Jira ID and chat programs can link to it 
and bots can bring up various details.  I don’t think a hash id for a mailing 
list is as simple or memorable.

A feature of a mailing list thread is that it can go in different directions 
easily.  The burden is that it will be harder to follow in the future if you’re 
trying to sort out implementation details.  So for high level discussion, the 
mailing list is great.  When getting down to the actual work and discussion 
about that focused work, that’s where a tool like Jira comes in.  Then it is 
reference-able in the changes.txt and other things.

I think the approach proposed by Jonathan is a nice way to keep dev list 
followers informed but keeping ticket details focused.

> On Aug 15, 2016, at 1:12 PM, Chris Mattmann  wrote:
> 
> How is it harder to point someone to mail?
> 
> Have you seen lists.apache.org?
> 
> Specifically:
> https://lists.apache.org/list.html?dev@cassandra.apache.org
> 
> 
> 
> On 8/15/16, 10:08 AM, "Jeremiah D Jordan"  wrote:
> 
>I like keeping things in JIRA because then everything is in one place, and 
> it is easy to refer someone to it in the future.
>But I agree that JIRA tickets with a bunch of design discussion and POC’s 
> and such in them can get pretty long and convoluted.
> 
>I don’t really like the idea of moving all of that discussion to email 
> which makes it has harder to point someone to it.  Maybe a better idea would 
> be to have a “design/POC” JIRA and an “implementation” JIRA.  That way we 
> could still keep things in JIRA, but the final decision would be kept “clean”.
> 
>Though it would be nice if people would send an email to the dev list when 
> proposing “design” JIRA’s, as not everyone has time to follow every JIRA ever 
> made to see that a new design JIRA was created that they might be interested 
> in participating on.
> 
>My 2c.
> 
>-Jeremiah
> 
> 
>> On Aug 15, 2016, at 9:22 AM, Jonathan Ellis  wrote:
>> 
>> A long time ago, I was a proponent of keeping most development discussions
>> on Jira, where tickets can be self contained and the threadless nature
>> helps keep discussions from getting sidetracked.
>> 
>> But Cassandra was a lot smaller then, and as we've grown it has become
>> necessary to separate out the signal (discussions of new features and major
>> changes) from the noise of routine bug reports.
>> 
>> I propose that we take advantage of the dev list to perform that
>> separation.  Major new features and architectural improvements should be
>> discussed first here, then when consensus on design is achieved, moved to
>> Jira for implementation and review.
>> 
>> I think this will also help with the problem when the initial idea proves
>> to be unworkable and gets revised substantially later after much
>> discussion.  It can be difficult to figure out what the conclusion was, as
>> review comments start to pile up afterwards.  Having that discussion on the
>> list, and summarizing on Jira, would mitigate this.
>> 
>> -- 
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder, http://www.datastax.com
>> @spyced
> 
> 
> 
> 

Re: Cassandra Java Driver and DataStax

[Jeremy Hanna]

> On Jun 5, 2016, at 4:33 PM, Mattmann, Chris A (3980) 
>  wrote:
> 
> Thanks for the info Jonathan. I think have assessed based on
> the replies thus far, my studying of the archives and
> commit and project history the following situation.
> 
> Unfortunately it seems like there is a bit of control going on
> I’m going to call a spade a spade here. A key portion of your 
> software’s stack, a client driver to use it, exists outside of
> Apache in separate communities. This is an inherent risk to the
> project. Some of you cite flexibility and adaptability as reasons
> for this - I’ve seen it in so many communities over the last 12+
> years in the foundation - it’s not really due to those issues.

Not all open-source projects do well under the apache umbrella in my opinion.  
Additionally not all library dependencies for all apache projects come from 
apache.

> There is definitely some control going on.

One thing I like about the ASF is that it’s about contribution and meritocracy. 
 If you have a company that is devoted to making a project successful, you’ll 
have more contribution from them.  Some people will gravitate to work for that 
company because they are passionate about the project and working there allows 
them to spend more time on it than they would have been able to at other 
companies.  And yet there are several committers and PMC members that don’t 
work for DataStax who have an influence over its development.  I think you may 
mean control in terms of contribution as you talk about in your next questions. 
 If that’s the case, how do you get other people to contribute more?  DataStax 
has already sponsored several contributor bootcamps for instance.  If it’s 
about contribution and meritocracy, is there an instance where a contribution 
was not accepted because of where an individual was employed?  Is there an 
instance where someone wasn’t accepted as a contributor or committer or pmc 
member because of where they worked?  Several of those committers/PMC members 
who are currently at DataStax became committers/PMC members before joining 
DataStax.  I’m just trying to understand the nature of where you see a problem.

> I would ask you all
> this - has there been a PR or patch in the past year or two that
> wasn’t singularly reviewed by DataStax committers and PMC? Also,
> as to the composition of the PMC when was the last time a non 
> DataStax person was elected to the PMC and/or as a committer?
> 
> By itself the diversity issues alone are not damning to the 
> project, but taken together with the citation to other project
> communities even those outside of Apache (e.g., the comments
> well “Postgres does it this way, so it’s a good example to
> compare us to” or “these other 4 projects at the ASF do it 
> like this, so X”.. [sic]) and with the perception being created
> to those that don’t work at DataStax, and there is an issue here.

I don’t quite understand the thinking here - referencing how successful 
open-source projects operate (outside the asf) is damning?  Mentioning that 
Cassandra is not unique within Apache to have client drivers outside of the 
main project is damning?  I don’t understand why that either of those would be 
in any way negative or invalid.

Regarding the thread generally, I still haven’t seen 1) an instance where 
having client drivers developed outside of the core project being a problem or 
2) an example where employees of datastax exerted control over the project to 
the detriment of others or 3) an example of anyone in the community saying 
“yeah, you’re right, they do control stuff and it sucks.”

Personally, I would like to see more specific concrete evidence of a problem.  
I’ve been involved with the project since 0.6 and it’s always had a very open 
and active community complete with ideas, disagreements, and mistakes.  Take 
for example CASSANDRA-9666 where the best time series compaction strategy 
alternatives are discussed.  It was determined that a community contribution 
from a third party company was going to supersede/replace what was in the tree. 
 If there are specific concerns, I think everyone involved in the project would 
like to know.

> 
> I would like to see a discussion in your next board report about
> the diversity and health issues of the project, and also some 
> ideas about potential strategies for mitigation. 
> 
> I appreciate the open and honest conversation thus far. Let’s
> keep it up.
> 
> Cheers,
> Chris
> 
> ++
> Chris Mattmann, Ph.D.
> Chief Architect
> Instrument Software and Science Data Systems Section (398)
> NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA
> Office: 168-519, Mailstop: 168-527
> Email: chris.a.mattm...@nasa.gov
> WWW:  http://sunset.usc.edu/~mattmann/
> ++
> Director, Information Retrieval and Data Science Group (IRDS)
> Adjunct 

Re: Short column names

[Jeremy Hanna]

It has been talked about in the past, see 
https://issues.apache.org/jira/browse/CASSANDRA-4175 for example.  However with 
https://issues.apache.org/jira/browse/CASSANDRA-8099, the duplication of column 
names is gone.  So once you’re on Cassandra 3+, this optimization is a lot less 
valuable.

> On Feb 11, 2016, at 4:19 PM, Bhuvan Rawal  wrote:
> 
> Hi,
> 
> We are modelling schema for database revamp from mysql to Cassandra. It has
> been recommended in several places that column names must be kept as small
> as possible to optimise disk storage.
> 
> I have a doubt here, why can't we map column names and store it as an
> index, say in memory. I mean, make column name really small human
> unreadable and store it in disk  but map it with real column while
> querying. That way one can go ahead with readable column names .
> 
> Let me know if I can go ahead and create a jira for the same
> 
> Regards,
> Bhuvan



signature.asc
Description: Message signed with OpenPGP using GPGMail

Re: Requiring Java 8 for C* 3.0

[Jeremy Hanna]

There’s no reason why people can’t run java 8 with 2.1.  IIRC the only issue 
we’d had with it was Dave’s 
https://issues.apache.org/jira/browse/CASSANDRA-7028.  That’s probably the best 
thing for people to do though - run java 8 with 2.1 so the jump to 3.0 isn’t as 
significant.  Good point.

 On May 7, 2015, at 11:43 AM, Nick Bailey n...@datastax.com wrote:
 
 Is running 2.1 with java 8 a supported or recommended way to run at this
 point? If not then we'll be requiring users to upgrade both java and C* at
 the same time when making the jump to 3.0.
 
 On Thu, May 7, 2015 at 11:25 AM, Aleksey Yeschenko alek...@apache.org
 wrote:
 
 The switch will necessarily hurt 3.0 adoption, but I think we’ll live. To
 me, the benefits (mostly access to lambdas and default methods, tbh)
 slightly outweigh the downsides.
 
 +0.1
 
 --
 AY
 
 On May 7, 2015 at 19:22:53, Gary Dusbabek (gdusba...@gmail.com) wrote:
 
 +1
 
 On Thu, May 7, 2015 at 11:09 AM, Jonathan Ellis jbel...@gmail.com wrote:
 
 We discussed requiring Java 8 previously and decided to remain Java
 7-compatible, but at the time we were planning to release 3.0 before
 Java 7
 EOL. Now that 8099 and increased emphasis on QA have delayed us past Java
 7 EOL, I think it's worth reopening this discussion.
 
 If we require 8, then we can use lambdas, LongAdder, StampedLock,
 Streaming
 collections, default methods, etc. Not just in 3.0 but over 3.x for the
 next year.
 
 If we don't, then people can choose whether to deploy on 7 or 8 -- but
 the
 vast majority will deploy on 8 simply because 7 is no longer supported
 without a premium contract with Oracle. 8 also has a more advanced G1GC
 implementation (see CASSANDRA-7486).
 
 I think that gaining access to the new features in 8 as we develop 3.x is
 worth losing the ability to run on a platform that will have been EOL
 for a
 couple months by the time we release.
 
 --
 Jonathan Ellis
 Project Chair, Apache Cassandra
 co-founder, http://www.datastax.com
 @spyced
 
 

Re: Refactoring cassandra service package

[Jeremy Hanna]

There was some hope started in CASSANDRA-6881 - see some of the later comments: 
https://issues.apache.org/jira/browse/CASSANDRA-6881

On 4 Jun 2014, at 04:04, Brian O'Neill b...@alumni.brown.edu wrote:

 
 Interesting proposition.  We¹ve embedded Cassandra a few times, so I¹d be
 interested in an approach that makes that easier.
 
 Is there a way to do it incrementally?  Introduce the injection framework,
 and convert a few classes (those required for startup), then slowly
 convert the remainder?
 
 peanut gallery,
 -brian
 
 ---
 Brian O'Neill
 Chief Technology Officer
 
 Health Market Science
 The Science of Better Results
 2700 Horizon Drive € King of Prussia, PA € 19406
 M: 215.588.6024 € @boneill42 http://www.twitter.com/boneill42  €
 healthmarketscience.com
 
 This information transmitted in this email message is for the intended
 recipient only and may contain confidential and/or privileged material. If
 you received this email in error and are not the intended recipient, or
 the person responsible to deliver it to the intended recipient, please
 contact the sender at the email above and delete this email and any
 attachments and destroy any copies thereof. Any review, retransmission,
 dissemination, copying or other use of, or taking any action in reliance
 upon, this information by persons or entities other than the intended
 recipient is strictly prohibited.
 
 
 
 
 
 
 
 On 6/3/14, 1:59 PM, Gary Dusbabek gdusba...@gmail.com wrote:
 
 On Tue, Jun 3, 2014 at 3:52 AM, Simon Chemouil schemo...@gmail.com
 wrote:
 
 Hi,
 
 I'm new to Cassandra and felt like exploring and hacking on the code. I
 was surprised to see the usage of so many mutable/global state statics
 all over the service package (basically global variables/singletons).
 
 While I understand it can be practical to work with singletons, and that
 in any case I'm not sure multi-tenant Cassandra (as in two different
 Cassandra instances within the same process) would make sense at all (or
 even work considering there is some native access going on with JNA), I
 find static state can easily lead to tangled 'spaghetti' code (accessing
 the singletons from anywhere, even where one shouldn't), and in general
 it ties the code to the VM instance, rather than to the class.
 
 I tried to find if it was an actual design choice, but from my
 understanding this is more something inherited from the early Cassandra
 times at Facebook. I just found this thread[1] pointing to issue
 CASSANDRA-741 (slightly more limited scope) that was marked as WONTFIX
 because no one took it (but still marked as open for contribution). The
 current code conventions also don't mention the usage of singletons
 except by stating:  Do not extract interfaces (or abstract classes)
 unless you actually need multiple implementations of it (switching to a
 service-style design doesn't require passing interfaces but it's
 highly encouraged to help testability).
 
 So, I'd like to try to make this refactoring happen and remove all (or
 most) mutable static state. It would be an easy way in for me in
 Cassandra's internals (maybe to contribute further). I think it would
 help testing (ability to unit test components without going to the
 storage for instance) and in general modernize the code. It would also
 make hacking on Cassandra easier because people could pick different
 pieces without pulling the whole thing.
 
 It would definitely break backwards compatibility with current Java code
 that directly embeds Cassandra / uses it as a library, but I would keep
 the same abstraction so the refactoring would be easy. In any case,
 backwards compatibility can be broken by many more changes than just
 refactoring, and once this is done it will be easier to deal with
 backwards compatibility.
 
 Obviously all .instance fields would be gone, and I'd try to fix
 potential cyclic class dependencies and generally make sure classes
 dependencies form a direct acyclic graph with CassandraDaemon as its
 root. The basic idea is to have each 'service' component require all its
 service dependencies in their constructor (and keeping them as a final
 field), rather than getting them via the global namespace (singleton
 instances).
 
 If I had it my way, I'd probably use a dependency injection framework,
 namely Dagger which is as far as I knpw the lightest Java DI framework
 actively developed (jointly developed by Square and Google's Java team
 responsible for Guice  Guava), which has a neat compile-time annotation
 processor that detects missing dependencies early on. It works with both
 Android and J2SE and is very fast, simple and light (65kB vs 710kB for
 Guice).
 
 So, the question is: would you guys accept such a patch? I'd rather not
 do the work if it has no chance of being merged upstream :).
 
 
 This has come up before. Let's face it, removing the singletons is a
 tempting proposition.
 
 Several of us have been down the path of trying to do it.
 
 At the end of the day, here's 

Re: Implementing a driver for cassandra native protocol v2

[Jeremy Hanna]

Also, you might see if you could join efforts with Aleksey on 
https://github.com/iamaleksey/seestar as well :)  Or at least exchange ideas.

On 28 Oct 2013, at 14:17, Sylvain Lebresne sylv...@datastax.com wrote:

 On Sat, Oct 26, 2013 at 8:07 PM, Mathieu D'Amours math...@damours.orgwrote:
 
 Hello,
 
 I'm currently implementing an Erlang driver for Cassandra's binary
 protocol v2. Most of it is straightforward, but I came across this part in
 [4.1.4. QUERY]:
 
 flags is a [byte] whose bits define the options for this query and
in particular influence what the remainder of the message contains.
A flag is set if the bit corresponding to its `mask` is set.
 Supported
flags are, given there mask:
  0x01 : ...
  0x02 : ...
  0x03 : ...
  0x04 : ...
 
 Does that mean that the 0x01 flag is set if `1  0` is set? and 0x03 is
 set if `1  2` is set?
 
 
 This is what it means, but really that's a typo. The mask values should
 read 0x01, 0x02, 0x04, 0x08 and 0x10. I'll fix the spec.
 
 --
 Sylvain
 
 
 
 Thanks in advance
 
 Mathieu

Re: Configuration of network connectors

[Jeremy Hanna]

Have you seen https://github.com/pcmanus/ccm as described in 
http://www.datastax.com/dev/blog/ccm-a-development-tool-for-creating-local-cassandra-clusters
 or does that not fit your use case?

On 9 Jul 2013, at 14:02, Łukasz Dywicki l...@code-house.org wrote:

 Hello,
 First of all I would like to say hello to cassandra user and developer 
 community. :)
 
 I write because we are using Cassandra in our unit tests and we have some 
 troubles with network connectivity. We ca not run multiple cassandra 
 instances during tests because we would need to randomize configuration of 
 port and so on. For now if we try to fork our tests we get address already 
 in use on one from two ports - native or thrift. In other apache projects we 
 can VM connectors (ActiveMQ, Camel, Mina) based on in-memory queue. I took 
 some time to see how CassandraDaemon starts servers and it's kinda of 
 hardcoded. I thought about changing configuration to be more like:
 
 servers:
  - class org.apache.cassandra.thrift.ThriftServer
  - class org.apache.cassandra.transport.Server
 
 Then we will be able to disable these servers for unit tests:
 servers:
  - class org.apache.cassandra.vm.VmServer
 
 This requires some small changes in daemon code and client libraries. I'm not 
 really deeply involved in cassandra stuff so I don't know the internal 
 architecture and implications thus I look forward for you to discuss this 
 topic.
 
 Cheers,
 Łukasz Dywicki
 --
 l...@code-house.org
 Twitter: ldywicki
 Blog: http://dywicki.pl
 Code-House - http://code-house.org