I'm tentatively in favor of the idea of us cutting releases for all our ecosystem dependencies as a blocker to cutting a major with Cassandra proper. I say tentatively since we've had trouble getting majors across the line for awhile so adding more dependencies to that feels risky, however I think the improvement to user experience justifies it.
Also - I suspect the effort required to get subprojects across the line will be quite a bit less than the mainline DB. If this is something there's agreement / consensus on, I'd be happy to take on the role of driving that coordination in the future. On Wed, Oct 2, 2024, at 3:16 PM, Jon Haddad wrote: > > Mostly Analytics, which should not be a blocker for Sidecar. > > Yes, agreed. I'm just trying to understand the context of the vnode > statement and how we're framing 1.0 sidecar. > > > We definitely need to support 5.0 for the Analytics release, but that's > > orthogonal to Sidecar. > > It is, unless we're endorsing the analytics library as a primary reason to > use sidecar. Then it becomes a dependency people rely on, and I don't want > it blocking people from upgrading. > > > Yeah, definitely. I agree that we need to support the latest released > > version of Cassandra in ecosystem projects. However, without official > > releases there won't be adoption and without adoption there won't be > > feedback from the community on what features / improvements are needed. > > I don't expect that our first release will be feature complete, but it should > be at least compelling. I'm still not aware of what functionality exists > that would meet that requirement. > > Think about this from the perspective of reading a blog post. We're excited > to announce sidecar 1.0! Here's what you can do: > > 1. Backup / restore? > 2. ? > 3. ? > > All I'm asking for are 3 reasons why people should care. If one of them is > backups, do we provide more flexible backup targets than S3, and if we > provide incremental / continuous backup options? Is there a scheduler or do > people need to roll their own? Is it coordinated, or is the intent that > people handle it on their own? > > I work with a lot of end users - this is the type of thing that affects > people and can either help or harm the project image. > > Jon > > > On Wed, Oct 2, 2024 at 2:40 PM Francisco Guerrero <fran...@apache.org> wrote: >> > By vnode support, do you mean the analytics library? Or do other features >> > in sidecar not work with vnodes? >> >> Mostly Analytics, which should not be a blocker for Sidecar. However, I do >> feel there should be more testing around vnodes in Sidecar. >> >> > If we're talking about analytics, that gets a little complicated. Are we >> > also then talking about 1.0'ing analytics? If so, I think we need support >> > for 5.0 and BTI there. >> >> We need to release Analytics at some point as well, we should have a similar >> discuss thread regarding Analytics. We definitely need to support 5.0 for the >> Analytics release, but that's orthogonal to Sidecar. >> >> > Increasing the size of the ecosystem is challenging... >> >> Yeah, definitely. I agree that we need to support the latest released version >> of Cassandra in ecosystem projects. However, without official releases >> there won't be adoption and without adoption there won't be feedback from >> the community on what features / improvements are needed. >> >> On 2024/10/02 18:05:42 Jon Haddad wrote: >> > By vnode support, do you mean the analytics library? Or do other features >> > in sidecar not work with vnodes? >> > >> > If we're talking about analytics, that gets a little complicated. Are we >> > also then talking about 1.0'ing analytics? If so, I think we need support >> > for 5.0 and BTI there. >> > >> > In my opinion, if something core to the project has official releases, such >> > as drivers or operational tooling that people depend on, it should also >> > support the latest version of Cassandra, on release. It doesn't look good >> > if we release C* without usable drivers or tooling to operate it. It is >> > massively deflating to announce we just released 5.0 (exciting!) but you >> > can't use it for 6 months because you're using analytics lib and the people >> > that contribute to it has better things to do with their time. I think >> > part of the obligation of maintaining these projects needs to be keeping up >> > with latest releases. If that can't be done, we should continue to treat >> > it as a use-as-your-own-risk thing without official releases. >> > >> > Increasing the size of the ecosystem is challenging... >> > >> > Jon >> > >> > >> > >> > >> > On Wed, Oct 2, 2024 at 1:21 PM Francisco Guerrero <fran...@apache.org> >> > wrote: >> > >> > > Hi folks, >> > > >> > > Thanks for all the input. I'm trying to gather all the comments, and from >> > > what I >> > > can gather it seems that most of the opinions are converging towards >> > > scoping >> > > a Sidecar release. These are the items that were called out that we will >> > > need >> > > for a release: >> > > - Documentation >> > > - Authorization / Authentication >> > > - vnode support >> > > >> > > There are some smaller bug fixes that need to be included that we can >> > > label >> > > as part of the 1.0 release.[1][2][3] >> > > >> > > If there are any other tasks we need to completed, I encourage the >> > > community >> > > to create JIRAs that can be in the release milestone for the Sidecar. >> > > >> > > I agree with Stefan that OpenAPI is desirable. OpenAPI is something I've >> > > been >> > > looking into as well. I'm also glad to see Abhijeet can help with the >> > > documentation effort, I can also help on that front. >> > > >> > > Hopefully, I've captured your comments as truly as possible. >> > > >> > > Thanks again for all the feedback. >> > > >> > > Best, >> > > - Francisco >> > > >> > > [1] https://issues.apache.org/jira/browse/CASSANDRASC-120 >> > > [2] https://issues.apache.org/jira/browse/CASSANDRASC-121 >> > > [3] https://issues.apache.org/jira/browse/CASSANDRASC-122 >> > > >> > > On 2024/10/02 15:29:20 Jon Haddad wrote: >> > > > When I developed some of the original sidecar code, it was based on >> > > > REST >> > > > Easy, which would have allowed us to generate the spec automatically. >> > > > I >> > > > did this in a similar project. >> > > > >> > > > That was removed here: >> > > > https://issues.apache.org/jira/browse/CASSANDRASC-57 >> > > > >> > > > But unfortunately it looks like you can't generate the spec from the >> > > code. >> > > > >> > > > Disappointing, that functionality was really useful. I wish someone >> > > > had >> > > > asked me before gutting it. >> > > > >> > > > Jon >> > > > >> > > > >> > > > >> > > > On Wed, Oct 2, 2024 at 11:16 AM Štefan Miklošovič < >> > > smikloso...@apache.org> >> > > > wrote: >> > > > >> > > > > Something like this: >> > > > > >> > > > > https://instaclustr.github.io/instaclustr-icarus-go-client/ >> > > > > >> > > > > On Wed, Oct 2, 2024 at 4:54 PM Štefan Miklošovič < >> > > smikloso...@apache.org> >> > > > > wrote: >> > > > > >> > > > >> While documenting endpoints please use something like OpenAPI >> > > > >> specification. The sidecar should expose this document itself so >> > > > >> when >> > > I go >> > > > >> to this and that URL, I see all endpoints, I put the payloads / >> > > parameters >> > > > >> for them and I can just directly call that, no messing with curl / >> > > wget or >> > > > >> programmatically or whatever like that. The barrier to exercise the >> > > basic >> > > > >> functionality has to virtually not be there. >> > > > >> >> > > > >> On Wed, Oct 2, 2024 at 4:13 PM Abhijeet Dubey < >> > > dubey.abhijee...@gmail.com> >> > > > >> wrote: >> > > > >> >> > > > >>> Hi folks, >> > > > >>> >> > > > >>> I have been using Sidecar recently and have found some of its >> > > > >>> functionalities to be quite useful. Hari and I are also working on >> > > CEP-40 >> > > > >>> which aims to introduce live migration features in Sidecar in the >> > > > >>> near future. >> > > > >>> >> > > > >>> However, as others have mentioned, I agree that it currently lacks >> > > > >>> proper documentation. >> > > > >>> >> > > > >>> Since this is an official Apache project, I believe that creating >> > > > >>> comprehensive documentation would be beneficial. This documentation >> > > should >> > > > >>> include an overview, architecture, a list and description of >> > > > >>> various >> > > > >>> endpoints, and some examples or tutorials on how to use Sidecar's >> > > features. >> > > > >>> >> > > > >>> This documentation would help people get started with Sidecar and >> > > lower >> > > > >>> the entry barrier for many. We can update the documentation >> > > incrementally >> > > > >>> as needed, along with future enhancements and new features. >> > > > >>> However, >> > > > >>> creating some form of formal documentation would be very helpful. >> > > > >>> >> > > > >>> To this end I'm willing and highly interested in writing some form >> > > > >>> of >> > > > >>> formal documentation for the Sidecar project. Please let me know >> > > > >>> your >> > > > >>> thoughts/opinions on this proposal. >> > > > >>> >> > > > >>> >> > > > >>> >> > > > >>> On Wed, Oct 2, 2024 at 6:46 PM Štefan Miklošovič < >> > > smikloso...@apache.org> >> > > > >>> wrote: >> > > > >>> >> > > > >>>> Totally agree with Jon here basically on all fronts. Apache >> > > Cassandra >> > > > >>>> Sidecar was always a hard nut to crack for me, that is probably >> > > > >>>> why >> > > I have >> > > > >>>> not been involved with that a lot even that is a great tool to >> > > > >>>> have >> > > and be >> > > > >>>> invested in as I was writing my own sidecar and I found a lot of >> > > > >>>> similarities and problems Apache's sidecar tries to fix. There was >> > > some >> > > > >>>> invisible barrier I have never managed to jump over. I was looking >> > > around >> > > > >>>> and I am very sorry if I just have not found it yet but there is >> > > not a list >> > > > >>>> of endpoints a sidecar has, is there? In readme and dev docs there >> > > is just >> > > > >>>> nothing. Taking it at a face value I just don't know what Sidecar >> > > > >>>> is >> > > > >>>> capable of and how to use it. I see in the commit history there is >> > > a bunch >> > > > >>>> of commits mentioning S3 but it is a total blackbox for me as a >> > > potential >> > > > >>>> user. >> > > > >>>> >> > > > >>>> On Wed, Oct 2, 2024 at 2:52 PM Jon Haddad >> > > > >>>> <j...@rustyrazorblade.com> >> > > > >>>> wrote: >> > > > >>>> >> > > > >>>>> I don't think we should release sidecar 1.0 without any docs. >> > > > >>>>> >> > > > >>>>> I took a look through the closed JIRAs to see what's there. >> > > > >>>>> Here's >> > > > >>>>> what I found, please correct me if there's more: >> > > > >>>>> >> > > > >>>>> - Lots of stuff related to analytics. >> > > > >>>>> >> > > > >>>>> I would be pretty excited for this, but the analytics library >> > > > >>>>> only >> > > > >>>>> works with single token clusters. Most folks don't run Cassandra >> > > this >> > > > >>>>> way. I realize there's some element of everyone needs to scratch >> > > their own >> > > > >>>>> itch, but I don't think we can really call this a useful feature >> > > if the >> > > > >>>>> overwhelming majority of folks can't use it. I've worked with a >> > > couple >> > > > >>>>> hundred teams over the years and can only think of 1 org outside >> > > of Apple >> > > > >>>>> and Netflix that used 1 token, and It was a cluster that predated >> > > v-nodes. >> > > > >>>>> >> > > > >>>>> The analytics repo says it's compatible with Cassandra 4, but not >> > > 5. >> > > > >>>>> >> > > > >>>>> - Backup & Restore from S3 >> > > > >>>>> >> > > > >>>>> Is this compatible with other cloud providers or object stores? >> > > > >>>>> It >> > > > >>>>> specifically lists S3 in JIRA. I haven't looked at the source >> > > yet. Am I >> > > > >>>>> correct in reading it supports backing up snapshots, no >> > > > >>>>> continuous >> > > > >>>>> backups? Seems like we should have at least feature parity with >> > > Medusa if >> > > > >>>>> we're going to release something here. >> > > > >>>>> >> > > > >>>>> All the other closed JIRAs look related to these two items. So >> > > > >>>>> the >> > > > >>>>> question is, are we releasing 1.0 as an limited S3 backup and >> > > restore >> > > > >>>>> tool? One that prevents you from upgrading to Cassandra 5 if you >> > > happen to >> > > > >>>>> use single token clusters? >> > > > >>>>> >> > > > >>>>> Who is the target audience? >> > > > >>>>> >> > > > >>>>> Jon >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> >> > > > >>>>> On Wed, Oct 2, 2024 at 2:41 AM Dinesh Joshi <djo...@apache.org> >> > > wrote: >> > > > >>>>> >> > > > >>>>>> Currently the Sidecar has a lot of functionality that is >> > > immediately >> > > > >>>>>> usable by the community. Apart from minor fixes, the AuthN/Z >> > > story would be >> > > > >>>>>> wrapped up soon. Post this, I would propose moving forward with >> > > cutting a >> > > > >>>>>> release with the existing feature set so we can get this in the >> > > hands of >> > > > >>>>>> our community. >> > > > >>>>>> >> > > > >>>>>> On Tue, Oct 1, 2024 at 8:27 PM guo Maxwell >> > > > >>>>>> <cclive1...@gmail.com> >> > > > >>>>>> wrote: >> > > > >>>>>> >> > > > >>>>>>> Have the same question ： what ‘s the plan ？ >> > > > >>>>>>> >> > > > >>>>>>> Jeff Jirsa <jji...@gmail.com>于2024年10月2日 周三上午10:43写道： >> > > > >>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> On Oct 1, 2024, at 7:26 PM, Josh McKenzie >> > > > >>>>>>>> <jmcken...@apache.org >> > > > >> > > > >>>>>>>> wrote: >> > > > >>>>>>>> >> > > > >>>>>>>> However it is used by a number of other features as a >> > > > >>>>>>>> dependency >> > > > >>>>>>>> such as analytics, backup/restore, repair, metrics, and CDC >> > > > >>>>>>>> >> > > > >>>>>>>> It seems like a natural pressure relief valve for moving >> > > operations >> > > > >>>>>>>> out of a core C* node that are well served out of process. >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> Yea, but the point of the foundation is to RELEASE software >> > > > >>>>>>>> for >> > > the >> > > > >>>>>>>> public good, and the link asserting consensus was dec2018, so >> > > its’ 5.5 >> > > > >>>>>>>> years and no releases. >> > > > >>>>>>>> >> > > > >>>>>>>> What’s the plan here? >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>>>>>>> >> > > > >>> >> > > > >>> -- >> > > > >>> *Abhijeet* >> > > > >>> >> > > > >> >> > > > >> > > >> >
