On Thu, Apr 12, 2018 at 4:41 PM, Dinesh Joshi <dinesh.jo...@yahoo.com.invalid> wrote: > Hey all - > With the uptick in discussion around Cassandra operability and after > discussing potential solutions with various members of the community, we > would like to propose the addition of a management process/sub-project into > Apache Cassandra. The process would be responsible for common operational > tasks like bulk execution of nodetool commands, backup/restore, and health > checks, among others. We feel we have a proposal that will garner some > discussion and debate but is likely to reach consensus. > While the community, in large part, agrees that these features should exist > “in the database”, there is debate on how they should be implemented. > Primarily, whether or not to use an external process or build on > CassandraDaemon. This is an important architectural decision but we feel the > most critical aspect is not where the code runs but that the operator still > interacts with the notion of a single database. Multi-process databases are > as old as Postgres and continue to be common in newer systems like Druid. As > such, we propose a separate management process for the following reasons: > > - Resource isolation & Safety: Features in the management process will not > affect C*'s read/write path which is critical for stability. An isolated > process has several technical advantages including preventing use of > unnecessary dependencies in CassandraDaemon, separation of JVM resources like > thread pools and heap, and preventing bugs from adversely affecting the main > process. In particular, GC tuning can be done separately for the two > processes, hopefully helping to improve, or at least not adversely affect, > tail latencies of the main process. > > - Health Checks & Recovery: Currently users implement health checks in > their own sidecar process. Implementing them in the serving process does not > make sense because if the JVM running the CassandraDaemon goes south, the > healthchecks and potentially any recovery code may not be able to run. Having > a management process running in isolation opens up the possibility to not > only report the health of the C* process such as long GC pauses or stuck JVM > but also to recover from it. Having a list of basic health checks that are > tested with every C* release and officially supported will help boost > confidence in C* quality and make it easier to operate. > > - Reduced Risk: By having a separate Daemon we open the possibility to > contribute features that otherwise would not have been considered before eg. > a UI. A library that started many background threads and is operated > completely differently would likely be considered too risky for > CassandraDaemon but is a good candidate for the management process.
Makes sense IMO. > What can go into the management process? > - Features that are non-essential for serving reads & writes for eg. > Backup/Restore or Running Health Checks against the CassandraDaemon, etc. > > - Features that do not make the management process critical for > functioning of the serving process. In other words, if someone does not wish > to use this management process, they are free to disable it. > > We would like to initially build minimal set of features such as health > checks and bulk commands into the first iteration of the management process. > We would use the same software stack that is used to build the current > CassandraDaemon binary. This would be critical for sharing code between > CassandraDaemon & management processes. The code should live in-tree to make > this easy. > With regards to more in-depth features like repair scheduling and discussions > around compaction in or out of CassandraDaemon, while the management process > may be a suitable host, it is not our goal to decide that at this time. The > management process could be used in these cases, as they meet the criteria > above, but other technical/architectural reasons may exists for why it should > not be. > We are looking forward to your comments on our proposal, Sounds good to me. Personally, I'm a little less interested in things like health/availability checks and metrics collection, because there are already tools to solve this problem (and most places will already be using them). I'm more interested in things like cluster status, streaming, repair, etc. Something to automate/centralize database-specific command and control, and improve visibility. In-tree also makes sense (tools/ maybe?), but I would suggest working out of a branch initially, and seeking inclusion when there is something more concrete to discuss. -- Eric Evans john.eric.ev...@gmail.com --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
