First of all, I'd like to thank you all for the comments. It's crucial for me to get the feedback considering the fact I'm limited in discussing such details with anyone else :-)
We should start a new thread about the JMX deprecation for sure, but let me make a few comments here as well. In my past experience, we had some concerns using JMX on production environments: there were some difficulties integrating JMX with the internal role management security plugins, enabling JMX for exporting metrics most of the time means exposing management commands we'd like to avoid and maintaining an SSL-enabled JMX configuration requires additional attention outside the accepted security system as a whole. None of the above makes JMX a bad tool, but the valuable question here is - if there was an alternative tool that offered the same level of functionality, would the JMX tool still be used for production environments? Of course, let's not forget Dropwizard's metrics and the collection of reporters it offers (they do not recommend gathering metrics through production environments [1]). At the same time, as long as JMX maintenance remains simple, is limited in the number of classes it implements, and offers the same functionality as the other tools, it brings much more value to deployments, especially in development and/or test environments. This is one of the goals that I am trying to offer you in this enhancement proposal. This templatized solution over internal data collections is not limited only to Virtual Table (vtable), or JMX interfaces, I can imagine plugins providing any other access points (REST API, etc.) based on it. Responding to design comments: > why not just use the vtable logic? Actually, I have some concerns here as well. I even found the example in the source code [2], but in my opinion, this approach has some drawbacks: - It may have additional overhead parsing query statements; - Such queries may flood the user's query history, so we have to handle it somehow; - JMX (or other interfaces) becomes tightly coupled to the vtable implementation, making it difficult to change the latter; - vtable has its own drawbacks, like building table metadata and building the result set with the same column types requires some level of abstraction to not repeat this for every table; Considering everything above I came to a conclusion to create a SystemView and ViewRow abstraction (probably the naming is not my best talent). > If we add a column to the end of the JMX row did we just break users? To be honest, I didn't pay much attention to it, thinking it would be safe to add a new regular column. As far as the current state of virtual tables is concerned, there's nothing new here, but I'll invest my time in it once we agree (and if) on the high-level details. [1] https://metrics.dropwizard.io/4.2.0/manual/core.html#jmx [2] https://github.com/apache/cassandra/blob/trunk/src/java/org/apache/cassandra/db/SystemKeyspace.java#L614 On Mon, 30 Jan 2023 at 14:12, Claude Warren, Jr via dev <dev@cassandra.apache.org> wrote: > > Actually, Maxim's proposal does not depend on JMX being present or not. What > the proposal does is make it easier to create/sync multiple presentations of > the same internal data: Virtual Tables, JMX, Metrics, next year's greatest > data presentation strategy. Removing JMX from the mix just reduces the > number of implementations, it does not in any way invalidate or change the > proposal. > > On Mon, Jan 30, 2023 at 11:27 AM Ekaterina Dimitrova <e.dimitr...@gmail.com> > wrote: >> >> +1 on starting that discussion, thank you for volunteering Benjamin! >> >> On Mon, 30 Jan 2023 at 4:59, Benjamin Lerer <b.le...@gmail.com> wrote: >>> >>> It seems to me that the question that we need to answer, before Maxim puts >>> more effort into this work, is: what is the future of JMX? >>> I agree that we have never been clear about that decision up to now. At >>> least now we have a reason to clarify that point. >>> >>> I can start a discussion about that if people agree? >>> >>> Le dim. 29 janv. 2023 à 10:14, Dinesh Joshi <djo...@apache.org> a écrit : >>>> >>>> I’m also very interested in this area. I quickly skimmed the proposal and >>>> IIUC it doesn’t call for moving away from JMX. Instead I think it’s making >>>> it easier to expose metrics over various interfaces. Maxim please correct >>>> me if I’m wrong in my understanding. >>>> >>>> I also second Josh’s point on JMX usage. I know it’s disabled in some >>>> deployments but it is not common practice in most places. >>>> >>>> Dinesh >>>> >>>> On Jan 28, 2023, at 4:10 PM, Josh McKenzie <jmcken...@apache.org> wrote: >>>> >>>> >>>> First off - thanks so much for putting in this effort Maxim! This is >>>> excellent work. >>>> >>>> Some thoughts on the CEP and responses in thread: >>>> >>>> Considering that JMX is usually not used and disabled in production >>>> environments for various performance and security reasons, the operator >>>> may not see the same picture from various of Dropwizard's metrics >>>> exporters and integrations as Cassandra's JMX metrics provide [1][2]. >>>> >>>> I don't think this assertion is true. Cassandra is running in a lot of >>>> places in the world, and JMX has been in this ecosystem for a long time; >>>> we need data that is basically impossible to get to claim "JMX is usually >>>> not used in C* environments in prod". >>>> >>>> I also wonder about if we should care about JMX? I know many wish to >>>> migrate (its going to be a very long time) away from JMX, so do we need a >>>> wrapper to make JMX and vtables consistent? >>>> >>>> If we can move away from a bespoke vtable or JMX based implementation and >>>> instead have a templatized solution each of these is generated from, that >>>> to me is the superior option. There's little harm in adding new JMX >>>> endpoints (or hell, other metrics framework integration?) as a byproduct >>>> of adding new vtable exposed metrics; we have the same maintenance >>>> obligation to them as we have to the vtables and if it generates from the >>>> same base data, we shouldn't have any further maintenance burden due to >>>> its presence right? >>>> >>>> we wish to move away from JMX >>>> >>>> I do, and you do, and many people do, but I don't believe all people on >>>> the project do. The last time this came up in slack the conclusion was >>>> "Josh should go draft a CEP to chart out a path to moving off JMX while >>>> maintaining backwards-compat w/existing JMX metrics for environments that >>>> are using them" (so I'm excited to see this CEP pop up before I got to it! >>>> ;)). Moving to a system that gives us a 0-cost way to keep JMX and vtable >>>> in sync over time on new metrics seems like a nice compromise for folks >>>> that have built out JMX-based maintenance infra on top of C*. Plus >>>> removing the boilerplate toil on vtables. win-win. >>>> >>>> If we add a column to the end of the JMX row did we just break users? >>>> >>>> I *think* this is arguably true for a vtable / CQL-based solution as well >>>> from the "you don't know how people are using your API" perspective. >>>> Unless we have clear guidelines about discretely selecting the columns you >>>> want from a vtable and trust users to follow them, if people have brittle >>>> greedy parsers pulling in all data from vtables we could very well break >>>> them as well by adding a new column right? Could be wrong here; I haven't >>>> written anything that consumes vtable metric data and maybe the obvious >>>> idiom in the face of that is robust in the presence of column addition. >>>> /shrug >>>> >>>> It's certainly more flexible and simpler to write to w/out detonating >>>> compared to JMX, but it's still an API we'd be revving. >>>> >>>> On Sat, Jan 28, 2023, at 4:24 PM, Ekaterina Dimitrova wrote: >>>> >>>> Overall I have similar thoughts and questions as David. >>>> >>>> I just wanted to add a reminder about this thread from last summer[1]. We >>>> already have issues with the alignment of JMX and Settings Virtual Table. >>>> I guess this is how Maxim got inspired to suggest this framework proposal >>>> which I want to thank him for! (I noticed he assigned CASSANDRA-15254) >>>> >>>> Not to open the Pandora box, but to me the most important thing here is to >>>> come into agreement about the future of JMX and what we will do or not as >>>> a community. Also, how much time people are able to invest. I guess this >>>> will influence any directions to be taken here. >>>> >>>> [1] >>>> https://lists.apache.org/thread/8mjcwdyqoobpvw2262bqmskkhs76pp69 >>>> >>>> >>>> On Thu, 26 Jan 2023 at 12:41, David Capwell <dcapw...@apple.com> wrote: >>>> >>>> I took a look and I see the result is an interface that looks like the >>>> vtable interface, that is then used by vtables and JMX? My first thought >>>> is why not just use the vtable logic? >>>> >>>> I also wonder about if we should care about JMX? I know many wish to >>>> migrate (its going to be a very long time) away from JMX, so do we need a >>>> wrapper to make JMX and vtables consistent? I am cool with something like >>>> the following >>>> >>>> registerWithJMX(jmxName, query(“SELECT * FROM system_views.streaming”)); >>>> >>>> >>>> So if we want to have a JMX view that matches the table then that’s cool >>>> by me, but one thing that has been brought up in reviews is backwards >>>> compatibility with regard to adding columns… If we add a column to the end >>>> of the JMX row did we just break users? >>>> >>>> Considering that JMX is usually not used and disabled in production >>>> environments for various performance and security reasons, the operator >>>> may not see the same picture from various of Dropwizard's metrics exporters >>>> >>>> If this is a real problem people are hitting, we can always add the >>>> ability to push metrics to common systems with a pluggable way to add >>>> non-standard solutions. Dropwizard already support this so would be low >>>> hanging fruit to address this. >>>> >>>> To make the proposed changes backwards compatible with the previous >>>> version of Cassandra, all MBeans and Virtual Tables we already have will >>>> remain unchanged >>>> >>>> >>>> If this is for new JMX endpoints moving forward, I am not sure of the >>>> benefit for the same reason listed above; we wish to move away from JMX >>>> >>>> On Jan 25, 2023, at 10:51 AM, Maxim Muzafarov <mmu...@apache.org> wrote: >>>> >>>> Hello Cassandra Community, >>>> >>>> >>>> I've been faced with a number of inconsistencies in the user APIs of >>>> the internal data collections representation exposed through the >>>> Cassandra monitoring interfaces that need to be fully aligned from an >>>> operator perspective. First of all, I'm highlighting JMX, Dropwizard >>>> Metrics, and Virtual Tables user interfaces. In order to address all >>>> these inconsistencies, I have created a draft enhancement proposal >>>> that describes everything I have found and how we can fix it once and >>>> for all. >>>> >>>> I'd like to hear your opinion and thoughts on it. Please take a look: >>>> https://docs.google.com/document/d/1j4J3bPWjQkAU9x4G-zxKObxPrKg36jLRT6xpUoNJa8Q >>>> >>>> >>>> -- >>>> Maxim Muzafarov >>>> >>>>
