What are we going to do with CEP-1? Cut and rescope in a new CEP or rewrite CEP-1?
On Mon, Oct 14, 2024 at 11:18 AM Francisco Guerrero <fran...@apache.org> wrote: > Hi folks, > > It was great meeting some of you in person at Community over Code where > we had a chance to discuss the Cassandra Sidecar project. A lot of > you expressed interest in having a release of Sidecar to start using the > existing features in the project: > > - C* adapters for versions 4.0, 4.1, 5.0 and trunk > - Cassandra Analytics support > - Restoring SSTables from S3-compatible object storage > - Mutual TLS authentication > - Role base access control > - Cluster health checks > - Observability > - Sidecar Client > > Some of the call-outs in this thread of things we need for a release: > > - Documentation > - Bug fixes [1][2][3] > > These are other things mentioned in the thread, where we would like help > from the community: > > - vNode support [4] > - Backup support [5] > - Bulk APIs [6] > > Once we have documentation and the bug fixes mentioned above, I will > start a thread vote for a release of Cassandra Sidecar 0.1 alpha. > > We want the community to benefit from the features already present in > Sidecar. The more community engagement we have, the more feature-rich > will become. > > Additionally, we can leverage easy-cass-stress to have Cassandra Sidecar > included in your Cassandra clusters, so we can easily start trying it out. > > Please let me know your thoughts on this. > > Best, > - Francisco > > [1] https://issues.apache.org/jira/browse/CASSANDRASC-120 > [2] https://issues.apache.org/jira/browse/CASSANDRASC-121 > [3] https://issues.apache.org/jira/browse/CASSANDRASC-122 > [4] https://issues.apache.org/jira/browse/CASSANDRASC-149 > [5] https://issues.apache.org/jira/browse/CASSANDRASC-148 > [6] https://issues.apache.org/jira/browse/CASSANDRASC-3 > > On 2024/10/03 12:05:14 Josh McKenzie wrote: > > I'm tentatively in favor of the idea of us cutting releases for all our > ecosystem dependencies as a blocker to cutting a major with Cassandra > proper. I say tentatively since we've had trouble getting majors across the > line for awhile so adding more dependencies to that feels risky, however I > think the improvement to user experience justifies it. > > > > Also - I suspect the effort required to get subprojects across the line > will be quite a bit less than the mainline DB. > > > > If this is something there's agreement / consensus on, I'd be happy to > take on the role of driving that coordination in the future. > > > > On Wed, Oct 2, 2024, at 3:16 PM, Jon Haddad wrote: > > > > Mostly Analytics, which should not be a blocker for Sidecar. > > > > > > Yes, agreed. I'm just trying to understand the context of the vnode > statement and how we're framing 1.0 sidecar. > > > > > > > We definitely need to support 5.0 for the Analytics release, but > that's orthogonal to Sidecar. > > > > > > It is, unless we're endorsing the analytics library as a primary > reason to use sidecar. Then it becomes a dependency people rely on, and I > don't want it blocking people from upgrading. > > > > > > > Yeah, definitely. I agree that we need to support the latest > released version of Cassandra in ecosystem projects. However, without > official releases there won't be adoption and without adoption there won't > be feedback from the community on what features / improvements are needed. > > > > > > I don't expect that our first release will be feature complete, but it > should be at least compelling. I'm still not aware of what functionality > exists that would meet that requirement. > > > > > > Think about this from the perspective of reading a blog post. We're > excited to announce sidecar 1.0! Here's what you can do: > > > > > > 1. Backup / restore? > > > 2. ? > > > 3. ? > > > > > > All I'm asking for are 3 reasons why people should care. If one of > them is backups, do we provide more flexible backup targets than S3, and if > we provide incremental / continuous backup options? Is there a scheduler > or do people need to roll their own? Is it coordinated, or is the intent > that people handle it on their own? > > > > > > I work with a lot of end users - this is the type of thing that > affects people and can either help or harm the project image. > > > > > > Jon > > > > > > > > > On Wed, Oct 2, 2024 at 2:40 PM Francisco Guerrero <fran...@apache.org> > wrote: > > >> > By vnode support, do you mean the analytics library? Or do other > features > > >> > in sidecar not work with vnodes? > > >> > > >> Mostly Analytics, which should not be a blocker for Sidecar. However, > I do > > >> feel there should be more testing around vnodes in Sidecar. > > >> > > >> > If we're talking about analytics, that gets a little complicated. > Are we > > >> > also then talking about 1.0'ing analytics? If so, I think we need > support > > >> > for 5.0 and BTI there. > > >> > > >> We need to release Analytics at some point as well, we should have a > similar > > >> discuss thread regarding Analytics. We definitely need to support 5.0 > for the > > >> Analytics release, but that's orthogonal to Sidecar. > > >> > > >> > Increasing the size of the ecosystem is challenging... > > >> > > >> Yeah, definitely. I agree that we need to support the latest released > version > > >> of Cassandra in ecosystem projects. However, without official releases > > >> there won't be adoption and without adoption there won't be feedback > from > > >> the community on what features / improvements are needed. > > >> > > >> On 2024/10/02 18:05:42 Jon Haddad wrote: > > >> > By vnode support, do you mean the analytics library? Or do other > features > > >> > in sidecar not work with vnodes? > > >> > > > >> > If we're talking about analytics, that gets a little complicated. > Are we > > >> > also then talking about 1.0'ing analytics? If so, I think we need > support > > >> > for 5.0 and BTI there. > > >> > > > >> > In my opinion, if something core to the project has official > releases, such > > >> > as drivers or operational tooling that people depend on, it should > also > > >> > support the latest version of Cassandra, on release. It doesn't > look good > > >> > if we release C* without usable drivers or tooling to operate it. > It is > > >> > massively deflating to announce we just released 5.0 (exciting!) > but you > > >> > can't use it for 6 months because you're using analytics lib and > the people > > >> > that contribute to it has better things to do with their time. I > think > > >> > part of the obligation of maintaining these projects needs to be > keeping up > > >> > with latest releases. If that can't be done, we should continue to > treat > > >> > it as a use-as-your-own-risk thing without official releases. > > >> > > > >> > Increasing the size of the ecosystem is challenging... > > >> > > > >> > Jon > > >> > > > >> > > > >> > > > >> > > > >> > On Wed, Oct 2, 2024 at 1:21 PM Francisco Guerrero < > fran...@apache.org> > > >> > wrote: > > >> > > > >> > > Hi folks, > > >> > > > > >> > > Thanks for all the input. I'm trying to gather all the comments, > and from > > >> > > what I > > >> > > can gather it seems that most of the opinions are converging > towards > > >> > > scoping > > >> > > a Sidecar release. These are the items that were called out that > we will > > >> > > need > > >> > > for a release: > > >> > > - Documentation > > >> > > - Authorization / Authentication > > >> > > - vnode support > > >> > > > > >> > > There are some smaller bug fixes that need to be included that we > can label > > >> > > as part of the 1.0 release.[1][2][3] > > >> > > > > >> > > If there are any other tasks we need to completed, I encourage the > > >> > > community > > >> > > to create JIRAs that can be in the release milestone for the > Sidecar. > > >> > > > > >> > > I agree with Stefan that OpenAPI is desirable. OpenAPI is > something I've > > >> > > been > > >> > > looking into as well. I'm also glad to see Abhijeet can help with > the > > >> > > documentation effort, I can also help on that front. > > >> > > > > >> > > Hopefully, I've captured your comments as truly as possible. > > >> > > > > >> > > Thanks again for all the feedback. > > >> > > > > >> > > Best, > > >> > > - Francisco > > >> > > > > >> > > [1] https://issues.apache.org/jira/browse/CASSANDRASC-120 > > >> > > [2] https://issues.apache.org/jira/browse/CASSANDRASC-121 > > >> > > [3] https://issues.apache.org/jira/browse/CASSANDRASC-122 > > >> > > > > >> > > On 2024/10/02 15:29:20 Jon Haddad wrote: > > >> > > > When I developed some of the original sidecar code, it was > based on REST > > >> > > > Easy, which would have allowed us to generate the spec > automatically. I > > >> > > > did this in a similar project. > > >> > > > > > >> > > > That was removed here: > > >> > > > https://issues.apache.org/jira/browse/CASSANDRASC-57 > > >> > > > > > >> > > > But unfortunately it looks like you can't generate the spec > from the > > >> > > code. > > >> > > > > > >> > > > Disappointing, that functionality was really useful. I wish > someone had > > >> > > > asked me before gutting it. > > >> > > > > > >> > > > Jon > > >> > > > > > >> > > > > > >> > > > > > >> > > > On Wed, Oct 2, 2024 at 11:16 AM Štefan Miklošovič < > > >> > > smikloso...@apache.org> > > >> > > > wrote: > > >> > > > > > >> > > > > Something like this: > > >> > > > > > > >> > > > > https://instaclustr.github.io/instaclustr-icarus-go-client/ > > >> > > > > > > >> > > > > On Wed, Oct 2, 2024 at 4:54 PM Štefan Miklošovič < > > >> > > smikloso...@apache.org> > > >> > > > > wrote: > > >> > > > > > > >> > > > >> While documenting endpoints please use something like OpenAPI > > >> > > > >> specification. The sidecar should expose this document > itself so when > > >> > > I go > > >> > > > >> to this and that URL, I see all endpoints, I put the > payloads / > > >> > > parameters > > >> > > > >> for them and I can just directly call that, no messing with > curl / > > >> > > wget or > > >> > > > >> programmatically or whatever like that. The barrier to > exercise the > > >> > > basic > > >> > > > >> functionality has to virtually not be there. > > >> > > > >> > > >> > > > >> On Wed, Oct 2, 2024 at 4:13 PM Abhijeet Dubey < > > >> > > dubey.abhijee...@gmail.com> > > >> > > > >> wrote: > > >> > > > >> > > >> > > > >>> Hi folks, > > >> > > > >>> > > >> > > > >>> I have been using Sidecar recently and have found some of > its > > >> > > > >>> functionalities to be quite useful. Hari and I are also > working on > > >> > > CEP-40 > > >> > > > >>> which aims to introduce live migration features in Sidecar > in the > > >> > > > >>> near future. > > >> > > > >>> > > >> > > > >>> However, as others have mentioned, I agree that it > currently lacks > > >> > > > >>> proper documentation. > > >> > > > >>> > > >> > > > >>> Since this is an official Apache project, I believe that > creating > > >> > > > >>> comprehensive documentation would be beneficial. This > documentation > > >> > > should > > >> > > > >>> include an overview, architecture, a list and description > of various > > >> > > > >>> endpoints, and some examples or tutorials on how to use > Sidecar's > > >> > > features. > > >> > > > >>> > > >> > > > >>> This documentation would help people get started with > Sidecar and > > >> > > lower > > >> > > > >>> the entry barrier for many. We can update the documentation > > >> > > incrementally > > >> > > > >>> as needed, along with future enhancements and new features. > However, > > >> > > > >>> creating some form of formal documentation would be very > helpful. > > >> > > > >>> > > >> > > > >>> To this end I'm willing and highly interested in writing > some form of > > >> > > > >>> formal documentation for the Sidecar project. Please let me > know your > > >> > > > >>> thoughts/opinions on this proposal. > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> > > >> > > > >>> On Wed, Oct 2, 2024 at 6:46 PM Štefan Miklošovič < > > >> > > smikloso...@apache.org> > > >> > > > >>> wrote: > > >> > > > >>> > > >> > > > >>>> Totally agree with Jon here basically on all fronts. Apache > > >> > > Cassandra > > >> > > > >>>> Sidecar was always a hard nut to crack for me, that is > probably why > > >> > > I have > > >> > > > >>>> not been involved with that a lot even that is a great > tool to have > > >> > > and be > > >> > > > >>>> invested in as I was writing my own sidecar and I found a > lot of > > >> > > > >>>> similarities and problems Apache's sidecar tries to fix. > There was > > >> > > some > > >> > > > >>>> invisible barrier I have never managed to jump over. I was > looking > > >> > > around > > >> > > > >>>> and I am very sorry if I just have not found it yet but > there is > > >> > > not a list > > >> > > > >>>> of endpoints a sidecar has, is there? In readme and dev > docs there > > >> > > is just > > >> > > > >>>> nothing. Taking it at a face value I just don't know what > Sidecar is > > >> > > > >>>> capable of and how to use it. I see in the commit history > there is > > >> > > a bunch > > >> > > > >>>> of commits mentioning S3 but it is a total blackbox for me > as a > > >> > > potential > > >> > > > >>>> user. > > >> > > > >>>> > > >> > > > >>>> On Wed, Oct 2, 2024 at 2:52 PM Jon Haddad < > j...@rustyrazorblade.com> > > >> > > > >>>> wrote: > > >> > > > >>>> > > >> > > > >>>>> I don't think we should release sidecar 1.0 without any > docs. > > >> > > > >>>>> > > >> > > > >>>>> I took a look through the closed JIRAs to see what's > there. Here's > > >> > > > >>>>> what I found, please correct me if there's more: > > >> > > > >>>>> > > >> > > > >>>>> - Lots of stuff related to analytics. > > >> > > > >>>>> > > >> > > > >>>>> I would be pretty excited for this, but the analytics > library only > > >> > > > >>>>> works with single token clusters. Most folks don't run > Cassandra > > >> > > this > > >> > > > >>>>> way. I realize there's some element of everyone needs to > scratch > > >> > > their own > > >> > > > >>>>> itch, but I don't think we can really call this a useful > feature > > >> > > if the > > >> > > > >>>>> overwhelming majority of folks can't use it. I've worked > with a > > >> > > couple > > >> > > > >>>>> hundred teams over the years and can only think of 1 org > outside > > >> > > of Apple > > >> > > > >>>>> and Netflix that used 1 token, and It was a cluster that > predated > > >> > > v-nodes. > > >> > > > >>>>> > > >> > > > >>>>> The analytics repo says it's compatible with Cassandra 4, > but not > > >> > > 5. > > >> > > > >>>>> > > >> > > > >>>>> - Backup & Restore from S3 > > >> > > > >>>>> > > >> > > > >>>>> Is this compatible with other cloud providers or object > stores? It > > >> > > > >>>>> specifically lists S3 in JIRA. I haven't looked at the > source > > >> > > yet. Am I > > >> > > > >>>>> correct in reading it supports backing up snapshots, no > continuous > > >> > > > >>>>> backups? Seems like we should have at least feature > parity with > > >> > > Medusa if > > >> > > > >>>>> we're going to release something here. > > >> > > > >>>>> > > >> > > > >>>>> All the other closed JIRAs look related to these two > items. So the > > >> > > > >>>>> question is, are we releasing 1.0 as an limited S3 backup > and > > >> > > restore > > >> > > > >>>>> tool? One that prevents you from upgrading to Cassandra > 5 if you > > >> > > happen to > > >> > > > >>>>> use single token clusters? > > >> > > > >>>>> > > >> > > > >>>>> Who is the target audience? > > >> > > > >>>>> > > >> > > > >>>>> Jon > > >> > > > >>>>> > > >> > > > >>>>> > > >> > > > >>>>> > > >> > > > >>>>> On Wed, Oct 2, 2024 at 2:41 AM Dinesh Joshi < > djo...@apache.org> > > >> > > wrote: > > >> > > > >>>>> > > >> > > > >>>>>> Currently the Sidecar has a lot of functionality that is > > >> > > immediately > > >> > > > >>>>>> usable by the community. Apart from minor fixes, the > AuthN/Z > > >> > > story would be > > >> > > > >>>>>> wrapped up soon. Post this, I would propose moving > forward with > > >> > > cutting a > > >> > > > >>>>>> release with the existing feature set so we can get this > in the > > >> > > hands of > > >> > > > >>>>>> our community. > > >> > > > >>>>>> > > >> > > > >>>>>> On Tue, Oct 1, 2024 at 8:27 PM guo Maxwell < > cclive1...@gmail.com> > > >> > > > >>>>>> wrote: > > >> > > > >>>>>> > > >> > > > >>>>>>> Have the same question ： what ‘s the plan ？ > > >> > > > >>>>>>> > > >> > > > >>>>>>> Jeff Jirsa <jji...@gmail.com>于2024年10月2日 周三上午10:43写道： > > >> > > > >>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> On Oct 1, 2024, at 7:26 PM, Josh McKenzie < > jmcken...@apache.org > > >> > > > > > >> > > > >>>>>>>> wrote: > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> However it is used by a number of other features as a > dependency > > >> > > > >>>>>>>> such as analytics, backup/restore, repair, metrics, > and CDC > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> It seems like a natural pressure relief valve for > moving > > >> > > operations > > >> > > > >>>>>>>> out of a core C* node that are well served out of > process. > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> Yea, but the point of the foundation is to RELEASE > software for > > >> > > the > > >> > > > >>>>>>>> public good, and the link asserting consensus was > dec2018, so > > >> > > its’ 5.5 > > >> > > > >>>>>>>> years and no releases. > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> What’s the plan here? > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>>>>>>> > > >> > > > >>> > > >> > > > >>> -- > > >> > > > >>> *Abhijeet* > > >> > > > >>> > > >> > > > >> > > >> > > > > > >> > > > > >> > > > >
