For anyone looking Dinesh made a ticket already: CASSANDRA-14395 <https://issues.apache.org/jira/browse/CASSANDRA-14395>
On 18 April 2018 at 18:14, Vinay Chella <vinaykumar...@gmail.com> wrote: > This is a good initiative. We have advocated for and run a sidecar for the > past 5+ years, and we've learned and improved it a lot. We look forward to > moving features from Priam (such as backup, HTTP -> JMX, etc) incrementally > to this sidecar as they make sense. > > > Thanks, > Vinay Chella > > On Fri, Apr 13, 2018 at 7:01 AM, Eric Evans <john.eric.ev...@gmail.com> > wrote: > > > On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi > > <dinesh.jo...@yahoo.com.invalid> wrote: > > > Hey all - > > > With the uptick in discussion around Cassandra operability and after > > discussing potential solutions with various members of the community, we > > would like to propose the addition of a management process/sub-project > into > > Apache Cassandra. The process would be responsible for common operational > > tasks like bulk execution of nodetool commands, backup/restore, and > health > > checks, among others. We feel we have a proposal that will garner some > > discussion and debate but is likely to reach consensus. > > > While the community, in large part, agrees that these features should > > exist “in the database”, there is debate on how they should be > implemented. > > Primarily, whether or not to use an external process or build on > > CassandraDaemon. This is an important architectural decision but we feel > > the most critical aspect is not where the code runs but that the operator > > still interacts with the notion of a single database. Multi-process > > databases are as old as Postgres and continue to be common in newer > systems > > like Druid. As such, we propose a separate management process for the > > following reasons: > > > > > > - Resource isolation & Safety: Features in the management process > > will not affect C*'s read/write path which is critical for stability. An > > isolated process has several technical advantages including preventing > use > > of unnecessary dependencies in CassandraDaemon, separation of JVM > resources > > like thread pools and heap, and preventing bugs from adversely affecting > > the main process. In particular, GC tuning can be done separately for the > > two processes, hopefully helping to improve, or at least not adversely > > affect, tail latencies of the main process. > > > > > > - Health Checks & Recovery: Currently users implement health checks > > in their own sidecar process. Implementing them in the serving process > does > > not make sense because if the JVM running the CassandraDaemon goes south, > > the healthchecks and potentially any recovery code may not be able to > run. > > Having a management process running in isolation opens up the possibility > > to not only report the health of the C* process such as long GC pauses or > > stuck JVM but also to recover from it. Having a list of basic health > checks > > that are tested with every C* release and officially supported will help > > boost confidence in C* quality and make it easier to operate. > > > > > > - Reduced Risk: By having a separate Daemon we open the possibility > > to contribute features that otherwise would not have been considered > before > > eg. a UI. A library that started many background threads and is operated > > completely differently would likely be considered too risky for > > CassandraDaemon but is a good candidate for the management process. > > > > Makes sense IMO. > > > > > What can go into the management process? > > > - Features that are non-essential for serving reads & writes for eg. > > Backup/Restore or Running Health Checks against the CassandraDaemon, etc. > > > > > > - Features that do not make the management process critical for > > functioning of the serving process. In other words, if someone does not > > wish to use this management process, they are free to disable it. > > > > > > We would like to initially build minimal set of features such as health > > checks and bulk commands into the first iteration of the management > > process. We would use the same software stack that is used to build the > > current CassandraDaemon binary. This would be critical for sharing code > > between CassandraDaemon & management processes. The code should live > > in-tree to make this easy. > > > With regards to more in-depth features like repair scheduling and > > discussions around compaction in or out of CassandraDaemon, while the > > management process may be a suitable host, it is not our goal to decide > > that at this time. The management process could be used in these cases, > as > > they meet the criteria above, but other technical/architectural reasons > may > > exists for why it should not be. > > > We are looking forward to your comments on our proposal, > > > > Sounds good to me. > > > > Personally, I'm a little less interested in things like > > health/availability checks and metrics collection, because there are > > already tools to solve this problem (and most places will already be > > using them). I'm more interested in things like cluster status, > > streaming, repair, etc. Something to automate/centralize > > database-specific command and control, and improve visibility. > > > > In-tree also makes sense (tools/ maybe?), but I would suggest working > > out of a branch initially, and seeking inclusion when there is > > something more concrete to discuss. > > > > > > -- > > Eric Evans > > john.eric.ev...@gmail.com > > > > --------------------------------------------------------------------- > > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >
