Jon, That sounds good. Let's make these commands rely on the settings virtual table and keep the initial changes as minimal as possible.
We've also scheduled a Cassandra Contributor Meeting on January 30th 2024, so I'll prepare some slides with everything we've got so far and try to prepare some drafts to demonstrate the design. https://cwiki.apache.org/confluence/display/CASSANDRA/Cassandra+Contributor+Meeting On Tue, 9 Jan 2024 at 00:55, Jon Haddad <j...@jonhaddad.com> wrote: > > It's great to see where this is going and thanks for the discussion on the ML. > > Personally, I think adding two new ways of accomplishing the same thing is a > net negative. It means we need more documentation and creates > inconsistencies across tools and users. The tradeoffs you've listed are > worth considering, but in my opinion adding 2 new ways to accomplish the same > thing hurts the project more than it helps. > > > - I'd like to see a symmetry between the JMX and CQL APIs, so that users > > will have a sense of the commands they are using and are less > likely to check the documentation; > > I've worked with a couple hundred teams and I can only think of a few who use > JMX directly. It's done very rarely. After 10 years, I still have to look > up the JMX syntax to do anything useful, especially if there's any quoting > involved. Power users might know a handful of JMX commands by heart, but I > suspect most have a handful of bash scripts they use instead, or have a > sidecar. I also think very few users will migrate their management code from > JMX to CQL, nor do I imagine we'll move our own tools until the > `disablebinary` problem is solved. > > > - It will be easier for us to move the nodetool from the jmx client that is > > used under the hood to an implementation based on a java-driver and use the > > CQL for the same; > > I can't imagine this would make a material difference. If someone's > rewriting a nodetool command, how much time will be spent replacing the JMX > call with a CQL one? Looking up a virtual table isn't going to be what > consumes someone's time in this process. Again, this won't be done without > solving `nodetool disablebinary`. > > > if we have cassandra-15254 merged, it will cost almost nothing to support > > the exec syntax for setting properties; > > My concern is more about the weird user experience of having two ways of > doing the same thing, less about the technical overhead of adding a second > implementation. I propose we start simple, see if any of the reasons you've > listed are actually a real problem, then if they are, address the issue in a > follow up. > > If I'm wrong, it sounds like it's fairly easy to add `exec` for changing > configs. If I'm right, we'll have two confusing syntaxes forever. It's a > lot easier to add something later than take it away. > > How does that sound? > > Jon > > > > > On Mon, Jan 8, 2024 at 7:55 PM Maxim Muzafarov <mmu...@apache.org> wrote: >> >> > Some operations will no doubt require a stored procedure syntax, but >> > perhaps it would be a good idea to split the work into two: >> >> These are exactly the first steps I have in mind: >> >> [Ready for review] >> Allow UPDATE on settings virtual table to change running configurations >> https://issues.apache.org/jira/browse/CASSANDRA-15254 >> >> This issue is specifically aimed at changing the configuration >> properties we are talking about (value is in yaml format): >> e.g. UPDATE system_views.settings SET compaction_throughput = 128Mb/s; >> >> [Ready for review] >> Expose all table metrics in virtual table >> https://issues.apache.org/jira/browse/CASSANDRA-14572 >> >> This is to observe the running configuration and all available metrics: >> e.g. select * from system_views.thread_pools; >> >> >> I hope both of the issues above will become part of the trunk branch >> before we move on to the CQL management commands. In this topic, I'd >> like to discuss the design of the CQL API, and gather feedback, so >> that I can prepare a draft of changes to look at without any >> surprises, and that's exactly what this discussion is about. >> >> >> cqlsh> UPDATE system.settings SET compaction_throughput = 128; >> cqlsh> exec setcompactionthroughput 128 >> >> I don't mind removing the exec command from the CQL command API which >> is intended to change settings. Personally, I see the second option as >> just an alias for the first command, and in fact, they will have the >> same implementation under the hood, so please consider the rationale >> below: >> >> - I'd like to see a symmetry between the JMX and CQL APIs, so that >> users will have a sense of the commands they are using and are less >> likely to check the documentation; >> - It will be easier for us to move the nodetool from the jmx client >> that is used under the hood to an implementation based on a >> java-driver and use the CQL for the same; >> - if we have cassandra-15254 merged, it will cost almost nothing to >> support the exec syntax for setting properties; >> >> On Mon, 8 Jan 2024 at 20:13, Jon Haddad <j...@jonhaddad.com> wrote: >> > >> > Ugh, I moved some stuff around and 2 paragraphs got merged that shouldn't >> > have been. >> > >> > I think there's no way we could rip out JMX, there's just too many >> > benefits to having it and effectively zero benefits to removing. >> > >> > Regarding disablebinary, part of me wonders if this is a bit of a hammer, >> > and what we really want is "disable binary for non-admins". I'm not sure >> > what the best path is to get there. The local unix socket might be the >> > easiest path as it allows us to disable network binary easily and still >> > allow local admins, and allows the OS to reject the incoming connections >> > vs passing that work onto a connection handler which would have to >> > evaluate whether or not the user can connect. If a node is already in a >> > bad spot requring disable binary, it's probably not a good idea to have it >> > get DDOS'ed as part of the remediation. >> > >> > Sorry for multiple emails. >> > >> > Jon >> > >> > On Mon, Jan 8, 2024 at 4:11 PM Jon Haddad <j...@jonhaddad.com> wrote: >> >> >> >> > Syntactically, if we’re updating settings like compaction throughput, I >> >> > would prefer to simply update a virtual settings table >> >> > e.g. UPDATE system.settings SET compaction_throughput = 128 >> >> >> >> I agree with this, sorry if that wasn't clear in my previous email. >> >> >> >> > Some operations will no doubt require a stored procedure syntax, >> >> >> >> The alternative to the stored procedure syntax is to have first class >> >> support for operations like REPAIR or COMPACT, which could be >> >> interesting. It might be a little nicer if the commands are first class >> >> citizens. I'm not sure what the downside would be besides adding >> >> complexity to the parser. I think I like the idea as it would allow for >> >> intuitive tab completion (REPAIR <tab>) and mentally fit in with the rest >> >> of the permission system, and be fairly obvious what permission relates >> >> to what action. >> >> >> >> cqlsh > GRANT INCREMENTAL REPAIR ON mykeyspace.mytable TO jon; >> >> >> >> I realize the ability to grant permissions could be done for the stored >> >> procedure syntax as well, but I think it's a bit more consistent to >> >> represent it the same way as DDL and probably better for the end user. >> >> >> >> Postgres seems to generally do admin stuff with SELECT function(): >> >> https://www.postgresql.org/docs/9.3/functions-admin.html. It feels a bit >> >> weird to me to use SELECT to do things like kill DB connections, but that >> >> might just be b/c it's not how I typically work with a database. VACUUM >> >> is a standalone command though. >> >> >> >> Curious to hear what people's thoughts are on this. >> >> >> >> > I would like to see us move to decentralised structured settings >> >> > management at the same time, so that we can set properties for the >> >> > whole cluster, or data centres, or individual nodes via the same >> >> > mechanism - all from any node in the cluster. I would be happy to help >> >> > out with this work, if time permits. >> >> >> >> This would be nice. Spinnaker has this feature and I found it to be very >> >> valuable at Netflix when making large changes. >> >> >> >> Regarding JMX - I think since it's about as close as we can get to "free" >> >> I don't really consider it to be additional overhead, a decent escape >> >> hatch, and I can't see us removing any functionality that most teams >> >> would consider critical. >> >> >> >> > We need something that's available for use before the node comes fully >> >> > online >> >> > Supporting backwards compat, especially for automated ops (i.e. >> >> > nodetool, JMX, etc), is crucial. Painful, but crucial. >> >> >> >> I think there's no way we could rip out JMX, there's just too many >> >> benefits to having it and effectively zero benefits to removing. Part of >> >> me wonders if this is a bit of a hammer, and what we really want is >> >> "disable binary for non-admins". I'm not sure what the best path is to >> >> get there. The local unix socket might be the easiest path as it allows >> >> us to disable network binary easily and still allow local admins, and >> >> allows the OS to reject the incoming connections vs passing that work >> >> onto a connection handler which would have to evaluate whether or not the >> >> user can connect. If a node is already in a bad spot requring disable >> >> binary, it's probably not a good idea to have it get DDOS'ed as part of >> >> the remediation. >> >> >> >> I think it's safe to say there's no appetite to remove JMX, at least not >> >> for anyone that would have to rework their entire admin control plane, >> >> plus whatever is out there in OSS provisioning tools like puppet / chef / >> >> etc that rely on JMX. I see no value whatsoever in removing it. >> >> >> >> I should probably have phrased my earlier email a bit differently. Maybe >> >> this is better: >> >> >> >> Fundamentally, I think it's better for the project if administration is >> >> fully supported over CQL in addition to JMX, without introducing a >> >> redundant third option, with the project's preference being CQL. >> >> >> >> >> >> On Mon, Jan 8, 2024 at 9:10 AM Benedict Elliott Smith >> >> <bened...@apache.org> wrote: >> >>> >> >>> Syntactically, if we’re updating settings like compaction throughput, I >> >>> would prefer to simply update a virtual settings table >> >>> >> >>> e.g. UPDATE system.settings SET compaction_throughput = 128 >> >>> >> >>> Some operations will no doubt require a stored procedure syntax, but >> >>> perhaps it would be a good idea to split the work into two: one part to >> >>> address settings like those above, and another for maintenance >> >>> operations such as triggering major compactions, repair and the like? >> >>> >> >>> I would like to see us move to decentralised structured settings >> >>> management at the same time, so that we can set properties for the whole >> >>> cluster, or data centres, or individual nodes via the same mechanism - >> >>> all from any node in the cluster. I would be happy to help out with this >> >>> work, if time permits. >> >>> >> >>> >> >>> On 8 Jan 2024, at 11:42, Josh McKenzie <jmcken...@apache.org> wrote: >> >>> >> >>> Fundamentally, I think it's better for the project if administration is >> >>> fully done over CQL and we have a consistent, single way of doing things. >> >>> >> >>> Strongly agree here. With 2 caveats: >> >>> >> >>> Supporting backwards compat, especially for automated ops (i.e. >> >>> nodetool, JMX, etc), is crucial. Painful, but crucial. >> >>> We need something that's available for use before the node comes fully >> >>> online; the point Jeff always brings up when we discuss moving away from >> >>> JMX. So long as we have some kind of "out-of-band" access to nodes or >> >>> accommodation for that, we should be good. >> >>> >> >>> For context on point 2, see slack: >> >>> https://the-asf.slack.com/archives/CK23JSY2K/p1688745128122749?thread_ts=1688662169.018449&cid=CK23JSY2K >> >>> >> >>> I point out that JMX works before and after the native protocol is >> >>> running (startup, shutdown, joining, leaving), and also it's semi-common >> >>> for us to disable the native protocol in certain circumstances, so at >> >>> the very least, we'd then need to implement a totally different cql >> >>> protocol interface just for administration, which nobody has committed >> >>> to building yet. >> >>> >> >>> >> >>> I think this is a solvable problem, and I think the benefits of having a >> >>> single, elegant way of interacting with a cluster and configuring it >> >>> justifies the investment for us as a project. Assuming someone has the >> >>> cycles to, you know, actually do the work. :D >> >>> >> >>> On Sun, Jan 7, 2024, at 10:41 PM, Jon Haddad wrote: >> >>> >> >>> I like the idea of the ability to execute certain commands via CQL, but >> >>> I think it only makes sense for the nodetool commands that cause an >> >>> action to take place, such as compact or repair. We already have >> >>> virtual tables, I don't think we need another layer to run informational >> >>> queries. I see little value in having the following (I'm using exec >> >>> here for simplicity): >> >>> >> >>> cqlsh> exec tpstats >> >>> >> >>> which returns a string in addition to: >> >>> >> >>> cqlsh> select * from system_views.thread_pools >> >>> >> >>> which returns structured data. >> >>> >> >>> I'd also rather see updatable configuration virtual tables instead of >> >>> >> >>> cqlsh> exec setcompactionthroughput 128 >> >>> >> >>> Fundamentally, I think it's better for the project if administration is >> >>> fully done over CQL and we have a consistent, single way of doing >> >>> things. I'm not dead set on it, I just think less is more in a lot of >> >>> situations, this being one of them. >> >>> >> >>> Jon >> >>> >> >>> >> >>> On Wed, Jan 3, 2024 at 2:56 PM Maxim Muzafarov <mmu...@apache.org> wrote: >> >>> >> >>> Happy New Year to everyone! I'd like to thank everyone for their >> >>> questions, because answering them forces us to move towards the right >> >>> solution, and I also like the ML discussions for the time they give to >> >>> investigate the code :-) >> >>> >> >>> I'm deliberately trying to limit the scope of the initial solution >> >>> (e.g. exclude the agent part) to keep the discussion short and clear, >> >>> but it's also important to have a glimpse of what we can do next once >> >>> we've finished with the topic. >> >>> >> >>> My view of the Command<> is that it is an abstraction in the broader >> >>> sense of an operation that can be performed on the local node, >> >>> involving one of a few internal components. This means that updating a >> >>> property in the settings virtual table via an update statement, or >> >>> executing e.g. the setconcurrentcompactors command are just aliases of >> >>> the same internal command via different APIs. Another example is the >> >>> netstats command, which simply aggregates the MessageService metrics >> >>> and returns them in a human-readable format (just another way of >> >>> looking at key-value metric pairs). More broadly, the command input is >> >>> Map<String, String> and String as the result (or List<String>). >> >>> >> >>> As Abe mentioned, Command and CommandRegistry should be largely based >> >>> on the nodetool command set at the beginning. We have a few options >> >>> for how we can initially construct command metadata during the >> >>> registry implementation (when moving command metadata from the >> >>> nodetool to the core part), so I'm planning to consult with the >> >>> command representations of the k8cassandra project in the way of any >> >>> further registry adoptions have zero problems (by writing a test >> >>> openapi registry exporter and comparing the representation results). >> >>> >> >>> So, the MVP is the following: >> >>> - Command >> >>> - CommandRegistry >> >>> - CQLCommandExporter >> >>> - JMXCommandExporter >> >>> - the nodetool uses the JMXCommandExporter >> >>> >> >>> >> >>> = Answers = >> >>> >> >>> > What do you have in mind specifically there? Do you plan on rewriting >> >>> > a brand new implementation which would be partially inspired by our >> >>> > agent? Or would the project integrate our agent code in-tree or as a >> >>> > dependency? >> >>> >> >>> Personally, I like the state of the k8ssandra project as it is now. My >> >>> understanding is that the server part of a database always lags behind >> >>> the client and sidecar parts in terms of the jdk version and the >> >>> features it provides. In contrast, sidecars should always be on top of >> >>> the market, so if we want to make an agent part in-tree, this should >> >>> be carefully considered for the flexibility which we may lose, as we >> >>> will not be able to change the agent part within the sidecar. The only >> >>> closest change I can see is that we can remove the interceptor part >> >>> once the CQL command interface is available. I suggest we move the >> >>> agent part to phase 2 and research it. wdyt? >> >>> >> >>> >> >>> > How are the results of the commands expressed to the CQL client? Since >> >>> > the command is being treated as CQL, I guess it will be rows, right? >> >>> > If yes, some of the nodetool commands output are a bit hierarchical in >> >>> > nature (e.g. cfstats, netstats etc...). How are these cases handled? >> >>> >> >>> I think the result of the execution should be a simple string (or set >> >>> of strings), which by its nature matches the nodetool output. I would >> >>> avoid building complex output or output schemas for now to simplify >> >>> the initial changes. >> >>> >> >>> >> >>> > Any changes expected at client/driver side? >> >>> >> >>> I'd like to keep the initial changes to a server part only, to avoid >> >>> scope inflation. For the driver part, I have checked the ExecutionInfo >> >>> interface provided by the java-driver, which should probably be used >> >>> as a command execution status holder. We'd like to have a unique >> >>> command execution id for each command that is executed on the node, so >> >>> the ExecutionInfo should probably hold such an id. Currently it has >> >>> the UUID getTracingId(), which is not well suited for our case and I >> >>> think further changes and follow-ups will be required here (including >> >>> the binary protocol, I think). >> >>> >> >>> >> >>> > The term COMMAND is a bit abstract I feel (subjective)... And I also >> >>> > feel the settings part is overlapping with virtual tables. >> >>> >> >>> I think we should keep the term Command as broad as it possible. As >> >>> long as we have a single implementation of a command, and the cost of >> >>> maintaining that piece of the source code is low, it's even better if >> >>> we have a few ways to achieve the same result using different APIs. >> >>> Personally, the only thing I would vote for is the separation of >> >>> command and metric terms (they shouldn't be mixed up). >> >>> >> >>> >> >>> > How are the responses of different operations expressed through the >> >>> > Command API? If the Command Registry Adapters depend upon the command >> >>> > metadata for invoking/validating the command, then I think there has >> >>> > to be a way for them to interpret the response format also, right? >> >>> >> >>> I'm not sure, that I've got the question correctly. Are you talking >> >>> about the command execution result schema and the validation of that >> >>> schema? >> >>> >> >>> For now, I see the interface as follows, the result of the execution >> >>> is a type that can be converted to the same string as the nodetool has >> >>> for the corresponding command (so that the outputs match): >> >>> >> >>> Command<A, R> >> >>> { >> >>> printResult(A argument, R result, Consumer<String> printer); >> >>> } >> >>> >> >>> On Tue, 5 Dec 2023 at 16:51, Abe Ratnofsky <a...@aber.io> wrote: >> >>> > >> >>> > Adding to Hari's comments: >> >>> > >> >>> > > Any changes expected at client/driver side? While using >> >>> > > JMX/nodetool, it is clear that the command/operations are getting >> >>> > > executed against which Cassandra node. But a client can connect to >> >>> > > multiple hosts and trigger queries, then how can it ensure that >> >>> > > commands are executed against the desired Cassandra instance? >> >>> > >> >>> > Clients are expected to set the node for the given CQL statement in >> >>> > cases like this; see docstring for example: >> >>> > https://github.com/apache/cassandra-java-driver/blob/4.x/core/src/main/java/com/datastax/oss/driver/api/core/cql/Statement.java#L124-L147 >> >>> > >> >>> > > The term COMMAND is a bit abstract I feel (subjective). Some of the >> >>> > > examples quoted are referring to updating settings (for example: >> >>> > > EXECUTE COMMAND setconcurrentcompactors WITH >> >>> > > concurrent_compactors=5;) and some are referring to operations. >> >>> > > Updating settings and running operations are considerably different >> >>> > > things. They may have to be handled in their own way. And I also >> >>> > > feel the settings part is overlapping with virtual tables. If >> >>> > > virtual tables support writes (at least the settings virtual table), >> >>> > > then settings can be updated using the virtual table itself. >> >>> > >> >>> > I agree with this - I actually think it would be clearer if this was >> >>> > referred to as nodetool, if the set of commands is going to be largely >> >>> > based on nodetool at the beginning. There is a lot of documentation >> >>> > online that references nodetool by name, and changing the nomenclature >> >>> > would make that existing documentation harder to understand. If a user >> >>> > can understand this as "nodetool, but better and over CQL not JMX" I >> >>> > think that's a clearer transition than a new concept of "commands". >> >>> > >> >>> > I understand that this proposal includes more than just nodetool, but >> >>> > there's a benefit to having a tool with a name, and a web search for >> >>> > "cassandra commands" is going to have more competition and ambiguity. >> >>> >> >>>
