> > I would love to see added something like “all guard rails must work in a > rolling fashion where some nodes do not have the latest guard rails”, but > also cool leaving this to JIRA.
Good point, I have added a line about the rolling behaviour. The cassandra.yaml file is pretty big already and configuration of > guardrails seems like a good candidate to modularize here. Could also be > done during implementation (or discussed then), so don't hold up voting on > me. :) The configuration is supposed to go under a specific section in cassandra.yaml. There was a colon character missed in the example, I have fixed it to reflect that all guardrail parameters are grouped in a single section. Nevertheless, we can always further discuss the details on how we organize the config during implementation. On Wed, 10 Nov 2021 at 20:16, Joshua McKenzie <jmcken...@apache.org> wrote: > Kind of a nit, but I advocate for us having a guardrails.yaml and updating > this line: >> >> *cassandra.yaml:* allows configuring individual guardrails at startup, >> being globally disabled by default. These guardrails will also be >> dynamically configurable through JMX and/or virtual tables. > > The cassandra.yaml file is pretty big already and configuration of > guardrails seems like a good candidate to modularize here. Could also be > done during implementation (or discussed then), so don't hold up voting on > me. :) > > On Wed, Nov 10, 2021 at 3:06 PM David Capwell <dcapw...@apple.com.invalid> > wrote: > >> I am cool with voting. I would love to see added something like “all >> guard rails must work in a rolling fashion where some nodes do not have the >> latest guard rails”, but also cool leaving this to JIRA. >> >> > On Nov 10, 2021, at 11:17 AM, Ekaterina Dimitrova < >> e.dimitr...@gmail.com> wrote: >> > >> > I am personally ready to give you my vote :-) >> > >> > On Wed, 10 Nov 2021 at 13:09, Andrés de la Peña <adelap...@apache.org> >> > wrote: >> > >> >> I think the updated CEP incorporates the feedback above, unless I'm >> missing >> >> something. Are we ready to start a vote? >> >> >> >> On Tue, 2 Nov 2021 at 15:54, Andrés de la Peña <adelap...@apache.org> >> >> wrote: >> >> >> >>> Under "Migrating existing cassandra.yaml warn/fail thresholds”, I >> >> recently >> >>>> added a few things which are basically guardrails, so should be >> >> included in >> >>>> this set; they are configured by track_warnings >> (coordinator_read_size, >> >>>> local_read_size, and row_index_size). With track_warnings I setup >> the >> >>>> plumbing to have read queries trigger warnings (or abort the query) >> to >> >> the >> >>>> client exists (under "Event logging" you mention "and also to the >> client >> >>>> connection when applicable”) and isn’t limited to the coordinator >> >>>> participating in the query (previous limitation for tombstone >> warnings). >> >>>> One thing I found which was problematic for track_warnings was that >> >>>> altering clients is annoying as java and python both ignore the error >> >>>> message we send (see >> >>>> >> >> >> https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131 >> >> ). >> >>>> We log client warnings (if enabled) but ignore any detailed error >> >> message >> >>>> received from the server; it would be good to talk about client >> >>>> integrations and how users are informed of issues in more detail. >> >>> >> >>> >> >>> I have updated the CEP to include those thresholds among the ones that >> >>> could be migrated once we have the guardrails framework ready. I have >> >> also >> >>> mentioned the usage of internal messaging to be able to propagate the >> >>> outcome of guardrails triggered on nodes that are not the coordinator, >> >> and >> >>> the need of making changes on drivers. >> >>> >> >>> What I meant by "and also to the client connection when applicable" is >> >>> that some guardrails can be applied to things that are nor necessarily >> >>> associated to a client connection, such as compaction. I have tried >> to be >> >>> more explicit about that. >> >>> >> >>> >> >>> On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña < >> a.penya.gar...@gmail.com >> >>> >> >>> wrote: >> >>> >> >>>> Being able to configure guardrails dynamically makes a lot of sense >> to >> >>>> me, I have updated the CEP to mention that. I think we don't need to >> >> decide >> >>>> yet whether it would be done through JMX and/or virtual tables. >> >>>> >> >>>> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas <sc...@paradoxica.net> >> >>>> wrote: >> >>>> >> >>>>> Re: "I think you all know my feels on JMX." – >> >>>>> >> >>>>> Super fair - I'd meant to speak in terms of desired outcome ("the >> >>>>> feature should be dynamically configurable at runtime") rather than >> >>>>> implementation ("this should be via JMX"). 👍 >> >>>>> >> >>>>> On Nov 1, 2021, at 1:24 PM, David Capwell >> <dcapw...@apple.com.INVALID> >> >>>>> wrote: >> >>>>> >> >>>>> >> >>>>> If anyone wants to bite off making >> >>>>> >> >> >> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java >> >>>>> < >> >>>>> >> >> >> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java >> >>> >> >>>>> support mutability then we get vtable support. I am cool with JMX >> >> and/or >> >>>>> vtable, to me its just more important to allow dynamic setting of >> these >> >>>>> configs. >> >>>>> >> >>>>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote: >> >>>>> >> >>>>> having them only configured via yaml seems like a bad outcome >> >>>>> >> >>>>> >> >>>>> +1 >> >>>>> >> >>>>> I would like to see us move towards configuration being driven >> through >> >>>>> virtual tables where possible, so that the whole cluster can be >> managed >> >>>>> from a single interface. Not sure if this is the right place to bite >> >> this >> >>>>> off, but perhaps? >> >>>>> >> >>>>> From: Jeff Jirsa <jji...@gmail.com> >> >>>>> Date: Monday, 1 November 2021 at 16:47 >> >>>>> To: Cassandra DEV <dev@cassandra.apache.org> >> >>>>> Subject: Re: [DISCUSS] CEP-3: Guardrails >> >>>>> Without bike-shedding too much, guardrails would be great, building >> >> them >> >>>>> into a more general purpose framework that limits various dangerous >> >>>>> things >> >>>>> would be fantastic. The CEP says that the guardrails should be >> distinct >> >>>>> from the capability restrictions ( >> >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't >> >> see >> >>>>> why >> >>>>> that needs to be the case. A system-level guardrail and a >> >> personal-level >> >>>>> guardrail are both restrictions, they just have different scopes, so >> >>>>> implement the restriction framework first, and allow the scopes to >> be >> >>>>> expanded as needed? >> >>>>> >> >>>>> Naming wise, I don't know that I'd actually surface these as >> >>>>> "guardrails", >> >>>>> but more as general "limits", and having them only configured via >> yaml >> >>>>> seems like a bad outcome >> >>>>> >> >>>>> >> >>>>> >> >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 >> >>>>> >> >>>>> >> >>>>> >> >>>>> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña < >> adelap...@apache.org >> >>> >> >>>>> wrote: >> >>>>> >> >>>>> Hi everyone, >> >>>>> >> >>>>> I'd like to start a discussion about Guardrails proposal: >> >>>>> >> >>>>> >> >>>>> >> >> >> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails >> >>>>> >> >>>>> Guardrails are an easy way to enforce system-wide soft and hard >> limits >> >> to >> >>>>> prevent anti-patterns of bad usage and in the long run make it not >> >>>>> possible >> >>>>> to severely degrade the performance of a node/cluster through user >> >>>>> actions >> >>>>> such as having too many secondary indexes, too large partitions, >> almost >> >>>>> full disks, etc. >> >>>>> >> >>>>> Thanks, >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >>>>> >> >> >> >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org >> For additional commands, e-mail: dev-h...@cassandra.apache.org >> >>
