This is not part of core database and a separate repo and so my impression is that this can continue to make progress. Also we can always make progress and not merge it till freeze is lifted.
Open to ideas/suggestions if someone thinks otherwise. > On Sep 22, 2018, at 03:13, kurt greaves <k...@instaclustr.com> wrote: > > Is this something we're moving ahead with despite the feature freeze? > > On Sat, 22 Sep 2018 at 08:32, dinesh.jo...@yahoo.com.INVALID > <dinesh.jo...@yahoo.com.invalid> wrote: > >> I have created a sub-task - CASSANDRA-14783. Could we get some feedback >> before we begin implementing anything? >> >> Dinesh >> >> On Thursday, September 20, 2018, 11:22:33 PM PDT, Dinesh Joshi < >> dinesh.jo...@yahoo.com.INVALID> wrote: >> >> I have updated the doc with a short paragraph providing the >> clarification. Sankalp's suggestion is already part of the doc. If there >> aren't further objections could we move this discussion over to the jira >> (CASSANDRA-14395)? >> >> Dinesh >> >>> On Sep 18, 2018, at 10:31 AM, sankalp kohli <kohlisank...@gmail.com> >> wrote: >>> >>> How about we start with a few basic features in side car. How about >> starting with this >>> 1. Bulk nodetool commands: User can curl any sidecar and be able to run >> a nodetool command in bulk across the cluster. >>> >> <sidecar>:<port>/bulk/nodetool/tablestats?arg0=keyspace_name.table_name&arg1=<if >> required> >>> >>> And later >>> 2: Health checks. >>> >>> On Thu, Sep 13, 2018 at 11:34 AM dinesh.jo...@yahoo.com.INVALID < >> dinesh.jo...@yahoo.com.invalid> wrote: >>> I will update the document to add that point. The document did not mean >> to serve as a design or architectural document but rather something that >> would spark a discussion on the idea. >>> Dinesh >>> >>> On Thursday, September 13, 2018, 10:59:34 AM PDT, Jonathan Haddad < >> j...@jonhaddad.com <mailto:j...@jonhaddad.com>> wrote: >>> >>> Most of the discussion and work was done off the mailing list - there's >> a >>> big risk involved when folks disappear for months at a time and resurface >>> with big pile of code plus an agenda that you failed to loop everyone in >>> on. In addition, by your own words the design document didn't accurately >>> describe what was being built. I don't write this to try to argue about >>> it, I just want to put some perspective for those of us that weren't part >>> of this discussion on a weekly basis over the last several months. Going >>> forward let's keep things on the ML so we can avoid confusion and >>> frustration for all parties. >>> >>> With that said - I think Blake made a really good point here and it's >>> helped me understand the scope of what's being built better. Looking at >> it >>> from a different perspective it doesn't seem like there's as much overlap >>> as I had initially thought. There's the machinery that runs certain >> tasks >>> (what Joey has been working on) and the user facing side of exposing that >>> information in management tool. >>> >>> I do appreciate (and like) the idea of not trying to boil the ocean, and >>> working on things incrementally. Putting a thin layer on top of >> Cassandra >>> that can perform cluster wide tasks does give us an opportunity to move >> in >>> the direction of a general purpose user-facing admin tool without >>> committing to trying to write the full stack all at once (or even make >>> decisions on it now). We do need a sensible way of doing rolling >> restarts >>> / scrubs / scheduling and Reaper wasn't built for that, and even though >> we >>> can add it I'm not sure if it's the best mechanism for the long term. >>> >>> So if your goal is to add maturity to the project by making cluster wide >>> tasks easier by providing a framework to build on top of, I'm in favor of >>> that and I don't see it as antithetical to what I had in mind with >> Reaper. >>> Rather, the two are more complementary than I had originally realized. >>> >>> Jon >>> >>> >>> >>> >>> On Thu, Sep 13, 2018 at 10:39 AM dinesh.jo...@yahoo.com.INVALID >>> <dinesh.jo...@yahoo.com <mailto:dinesh.jo...@yahoo.com>.invalid> wrote: >>> >>>> I have a few clarifications - >>>> The scope of the management process is not to simply run repair >>>> scheduling. Repair scheduling is one of the many features we could >>>> implement or adopt from existing sources. So could we please split the >>>> Management Process discussion and the repair scheduling? >>>> After re-reading the management process proposal, I see we missed to >>>> communicate a basic idea in the document. We wanted to take a pluggable >>>> approach to various activities that the management process could >> perform. >>>> This could accommodate different implementations of common activities >> such >>>> as repair. The management process would provide the basic framework >> and it >>>> would have default implementations for some of the basic activities. >> This >>>> would allow for speedier iteration cycles and keep things extensible. >>>> Turning to some questions that Jon and others have raised, when I +1, >> my >>>> intention is to fully contribute and stay with this community. That >> said, >>>> things feel rushed for some but for me it feels like analysis >> paralysis. >>>> We're looking for actionable feedback and to discuss the management >> process >>>> _not_ repair scheduling solutions. >>>> Thanks, >>>> Dinesh >>>> >>>> >>>> >>>> On Sep 12, 2018, at 6:24 PM, sankalp kohli <kohlisank...@gmail.com >> <mailto:kohlisank...@gmail.com>> wrote: >>>> Here is a list of open discussion points from the voting thread. I >> think >>>> some are already answered but I will still gather these questions here. >>>> >>>> From several people: >>>> 1. Vote is rushed and we need more time for discussion. >>>> >>>> From Sylvain >>>> 2. About the voting process...I think that was addressed by Jeff Jirsa >> and >>>> deserves a separate thread as it is not directly related to this >> thread. >>>> 3. Does the project need a side car. >>>> >>>> From Jonathan Haddad >>>> 4. Are people doing +1 willing to contribute >>>> >>>> From Jonathan Ellis >>>> 5. List of feature set, maturity, maintainer availability from Reaper >> or >>>> any other project being donated. >>>> >>>> Mick Semb Wever >>>> 6. We should not vote on these things and instead build consensus. >>>> >>>> Open Questions from this thread >>>> 7. What technical debts we are talking about in Reaper. Can someone >> give >>>> concrete examples. >>>> 8. What is the timeline of donating Reaper to Apache Cassandra. >>>> >>>> On Wed, Sep 12, 2018 at 3:49 PM sankalp kohli <kohlisank...@gmail.com >> <mailto:kohlisank...@gmail.com>> >>>> wrote: >>>> >>>> >>>> (Using this thread and not the vote thread intentionally) >>>> For folks talking about vote being rushed. I would use the email from >>>> Joseph to show this is not rushed. There was no email on this thread >> for 4 >>>> months until I pinged. >>>> >>>> >>>> Dec 2016: Vinay worked with Jon and Alex to try to collaborate on >> Reaper to >>>> come up with design goals for a repair scheduler that could work at >> Netflix >>>> scale. >>>> >>>> ~Feb 2017: Netflix believes that the fundamental design gaps prevented >> us >>>> from using Reaper as it relies heavily on remote JMX connections and >>>> central coordination. >>>> >>>> Sep. 2017: Vinay gives a lightning talk at NGCC about a highly >> available >>>> and distributed repair scheduling sidecar/tool. He is encouraged by >>>> multiple committers to build repair scheduling into the daemon itself >> and >>>> not as a sidecar so the database is truly eventually consistent. >>>> >>>> ~Jun. 2017 - Feb. 2018: Based on internal need and the positive >> feedback at >>>> NGCC, Vinay and myself prototype the distributed repair scheduler >> within >>>> Priam and roll it out at Netflix scale. >>>> >>>> Mar. 2018: I open a Jira (CASSANDRA-14346) along with a detailed 20 >> page >>>> design document for adding repair scheduling to the daemon itself and >> open >>>> the design up for feedback from the community. We get feedback from >> Alex, >>>> Blake, Nate, Stefan, and Mick. As far as I know there were zero >> proposals >>>> to contribute Reaper at this point. We hear the consensus that the >>>> community would prefer repair scheduling in a separate distributed >> sidecar >>>> rather than in the daemon itself and we re-work the design to match >> this >>>> consensus, re-aligning with our original proposal at NGCC. >>>> >>>> Apr 2018: Blake brings the discussion of repair scheduling to the dev >> list >>>> ( >>>> >>>> >>>> >> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E >> < >> https://lists.apache.org/thread.html/760fbef677f27aa5c2ab4c375c7efeb81304fea428deff986ba1c2eb@%3Cdev.cassandra.apache.org%3E >>> >>>> ). >>>> Many community members give positive feedback that we should solve it >> as >>>> part of Cassandra and there is still no mention of contributing Reaper >> at >>>> this point. The last message is my attempted summary giving context on >> how >>>> we want to take the best of all the sidecars (OpsCenter, Priam, >> Reaper) and >>>> ship them with Cassandra. >>>> >>>> Apr. 2018: Dinesh opens CASSANDRA-14395 along with a public design >> document >>>> for gathering feedback on a general management sidecar. Sankalp and >> Dinesh >>>> encourage Vinay and myself to kickstart that sidecar using the repair >>>> scheduler patch >>>> >>>> Apr 2018: Dinesh reaches out to the dev list ( >>>> >>>> >>>> >> https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E >> < >> https://lists.apache.org/thread.html/a098341efd8f344494bcd2761dba5125e971b59b1dd54f282ffda253@%3Cdev.cassandra.apache.org%3E >>> >>>> ) >>>> about the general management process to gain further feedback. All >> feedback >>>> remains positive as it is a potential place for multiple community >> members >>>> to contribute their various sidecar functionality. >>>> >>>> May-Jul 2017: Vinay and I work on creating a basic sidecar for running >> the >>>> repair scheduler based on the feedback from the community in >>>> CASSANDRA-14346 and CASSANDRA-14395 >>>> >>>> Jun 2018: I bump CASSANDRA-14346 indicating we're still working on >> this, >>>> nobody objects >>>> >>>> Jul 2018: Sankalp asks on the dev list if anyone has feature Jiras >> anyone >>>> needs review for before 4.0, I mention again that we've nearly got the >>>> basic sidecar and repair scheduling work done and will need help with >>>> review. No one responds. >>>> >>>> Aug 2018: We submit a patch that brings a basic distributed sidecar and >>>> robust distributed repair to Cassandra itself. Dinesh mentions that he >> will >>>> try to review. Now folks appear concerned about it being in tree and >>>> instead maybe it should go in a different repo all together. I don't >> think >>>> we have consensus on the repo choice yet. >>>> >>>> On Sun, Sep 9, 2018 at 9:13 AM sankalp kohli <kohlisank...@gmail.com >> <mailto:kohlisank...@gmail.com>> >>>> wrote: >>>> >>>> >>>> I agree with Jon and I think folks who are talking about tech debts in >>>> Reaper should elaborate with examples about these tech debts. Can we be >>>> more precise and list them down? I see it spread out over this long >> email >>>> thread!! >>>> >>>> On Sun, Sep 9, 2018 at 6:29 AM Elliott Sims <elli...@backblaze.com >> <mailto:elli...@backblaze.com>> >>>> wrote: >>>> >>>> >>>> A big one to add to your list there, IMO as a user: >>>> * API for determining detailed repair state (and history?). >> Essentially, >>>> something beyond just "Is some sort of repair running?" so that tools >>>> like >>>> Reaper can parallelize better. >>>> >>>> On Sun, Sep 9, 2018 at 8:30 AM, Stefan Podkowinski <s...@apache.org >> <mailto:s...@apache.org>> >>>> wrote: >>>> >>>> >>>> Does it have to be a single project with functionality provided by >>>> multiple plugins? Designing a plugin API at this point seems to be a >>>> >>>> bit >>>> >>>> early and comes with additional complexity around managing plugins in >>>> general. >>>> >>>> I was more thinking into the direction of: "what can we do to enable >>>> people to create any kind of side car or tooling solution?". Thinks >>>> >>>> like: >>>> >>>> >>>> Common cluster discovery and management API >>>> * Detect local Cassandra processes >>>> * Discover and receive events on cluster topology >>>> * Get assigned tokens for nodes >>>> * Read node configuration >>>> * Health checks (as already proposed) >>>> >>>> Any side cars should be easy to install on nodes that already run >>>> >>>> Cassandra >>>> >>>> * Scripts for packaging (tar, deb, rpm) >>>> * Templates for systemd support, optionally with auto-startup >>>> >>>> dependency >>>> >>>> on the Cassandra main process >>>> >>>> Integration testing >>>> * Provide basic testing framework for mocking cluster state and >>>> >>>> messages >>>> >>>> >>>> Support for other languages / avoid having to use JMX >>>> * JMX bridge (HTTP? gRPC?, already implemented in #14346?) >>>> >>>> Obviously the whole side car discussion is not moving into a direction >>>> everyone's happy with. Would it be an option to take a step back and >>>> start implementing such a tooling framework with scripts and libraries >>>> for the features described above, as a small GitHub project, instead of >>>> putting an existing side-car solution up for vote? If that would work >>>> and we get people collaborating on code shared between existing >>>> side-cars, then we could take the next step and think about either >>>> revisit the "official Cassandra side-car" topic, or add the created >>>> client tooling framework as official sub-project to the Cassandra >>>> project (maybe via Apache incubator). >>>> >>>> >>>> On 08.09.18 02:49, Joseph Lynch wrote: >>>> >>>> On Fri, Sep 7, 2018 at 5:03 PM Jonathan Haddad <j...@jonhaddad.com >> <mailto:j...@jonhaddad.com>> >>>> >>>> wrote: >>>> >>>> >>>> We haven’t even defined any requirements for an admin tool. It’s >>>> >>>> >>>> >>>> hard to >>>> >>>> >>>> >>>> make a case for anything without agreement on what we’re trying to >>>> >>>> >>>> build. >>>> >>>> >>>> >>>> >>>> We were/are trying to sketch out scope/requirements in the #14395 and >>>> #14346 tickets as well as their associated design documents. I think >>>> the general proposed direction is a distributed 1:1 management >>>> >>>> >>>> sidecar >>>> >>>> >>>> process similar in architecture to Netflix's Priam except explicitly >>>> built to be general and pluggable by anyone rather than tightly >>>> coupled to AWS. >>>> >>>> Dinesh, Vinay and I were aiming for low amounts of scope at first and >>>> take things in an iterative approach with just enough upfront design >>>> but not so much we are unable to make any progress at all. For >>>> >>>> >>>> example >>>> >>>> >>>> maybe something like: >>>> >>>> 1. Get a super simple and non controversial sidecar process that >>>> >>>> >>>> ships >>>> >>>> >>>> with Cassandra and exposes a lightweight HTTP interface to e.g. some >>>> basic JMX endpoints >>>> 2a. Add a pluggable execution engine for cron/oneshot/scheduled jobs >>>> with the basic interfaces and state store and such >>>> 2b. Start scoping and implementing the full HTTP interface, e.g. >>>> backup status, cluster health status, etc ... >>>> 3a. Start integrating implementations of the jobs from 2a such as >>>> snapshot, backup, cluster restart, daemon + sstable upgrade, repair, >>>> etc >>>> 3b. Start integrating UI components that pair with the HTTP interface >>>> >>>> from 2b >>>> >>>> 4. ?? Perhaps start unlocking next generation operations like moving >>>> "background" activities like compaction, streaming, repair etc into >>>> one or more sidecar contained processes to ensure the main daemon >>>> >>>> >>>> only >>>> >>>> >>>> handles read+write requests >>>> >>>> There are going to be a lot of questions to answer, and I think >>>> >>>> >>>> trying >>>> >>>> >>>> to answer them all up front will mean that we get nowhere or make >>>> unfortunate compromises that cripple the project from the start. If >>>> people think we need to do more design and discussion than we have >>>> been doing then we can spend more time on the design, but personally >>>> I'd rather start iterating on code and prove value incrementally. If >>>> it doesn't work out we won't release it GA to the community ... >>>> >>>> -Joey >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org <mailto: >> dev-unsubscr...@cassandra.apache.org> >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org >> <mailto:dev-h...@cassandra.apache.org> >>>> >>>> >>>> >>>> >>>> --------------------------------------------------------------------- >>>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org <mailto: >> dev-unsubscr...@cassandra.apache.org> >>>> For additional commands, e-mail: dev-h...@cassandra.apache.org >> <mailto:dev-h...@cassandra.apache.org> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>>> >>> >>> -- >>> Jon Haddad >>> http://www.rustyrazorblade.com <http://www.rustyrazorblade.com/> >>> twitter: rustyrazorblade >> --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
