Re: [DISCUSS] CEP-3: Guardrails
12:53, Andrés de la Peña <
>> a.penya.gar...@gmail.com
>> >>>
>> >>> wrote:
>> >>>
>> >>>> Being able to configure guardrails dynamically makes a lot of sense
>> to
>> >>>> me, I have updated the CEP to mention that. I think we don't need to
>> >> decide
>> >>>> yet whether it would be done through JMX and/or virtual tables.
>> >>>>
>> >>>> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas
>> >>>> wrote:
>> >>>>
>> >>>>> Re: "I think you all know my feels on JMX." –
>> >>>>>
>> >>>>> Super fair - I'd meant to speak in terms of desired outcome ("the
>> >>>>> feature should be dynamically configurable at runtime") rather than
>> >>>>> implementation ("this should be via JMX"). 👍
>> >>>>>
>> >>>>> On Nov 1, 2021, at 1:24 PM, David Capwell
>>
>> >>>>> wrote:
>> >>>>>
>> >>>>>
>> >>>>> If anyone wants to bite off making
>> >>>>>
>> >>
>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
>> >>>>> <
>> >>>>>
>> >>
>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
>> >>>
>> >>>>> support mutability then we get vtable support. I am cool with JMX
>> >> and/or
>> >>>>> vtable, to me its just more important to allow dynamic setting of
>> these
>> >>>>> configs.
>> >>>>>
>> >>>>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote:
>> >>>>>
>> >>>>> having them only configured via yaml seems like a bad outcome
>> >>>>>
>> >>>>>
>> >>>>> +1
>> >>>>>
>> >>>>> I would like to see us move towards configuration being driven
>> through
>> >>>>> virtual tables where possible, so that the whole cluster can be
>> managed
>> >>>>> from a single interface. Not sure if this is the right place to bite
>> >> this
>> >>>>> off, but perhaps?
>> >>>>>
>> >>>>> From: Jeff Jirsa
>> >>>>> Date: Monday, 1 November 2021 at 16:47
>> >>>>> To: Cassandra DEV
>> >>>>> Subject: Re: [DISCUSS] CEP-3: Guardrails
>> >>>>> Without bike-shedding too much, guardrails would be great, building
>> >> them
>> >>>>> into a more general purpose framework that limits various dangerous
>> >>>>> things
>> >>>>> would be fantastic. The CEP says that the guardrails should be
>> distinct
>> >>>>> from the capability restrictions (
>> >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't
>> >> see
>> >>>>> why
>> >>>>> that needs to be the case. A system-level guardrail and a
>> >> personal-level
>> >>>>> guardrail are both restrictions, they just have different scopes, so
>> >>>>> implement the restriction framework first, and allow the scopes to
>> be
>> >>>>> expanded as needed?
>> >>>>>
>> >>>>> Naming wise, I don't know that I'd actually surface these as
>> >>>>> "guardrails",
>> >>>>> but more as general "limits", and having them only configured via
>> yaml
>> >>>>> seems like a bad outcome
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña <
>> adelap...@apache.org
>> >>>
>> >>>>> wrote:
>> >>>>>
>> >>>>> Hi everyone,
>> >>>>>
>> >>>>> I'd like to start a discussion about Guardrails proposal:
>> >>>>>
>> >>>>>
>> >>>>>
>> >>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>> >>>>>
>> >>>>> Guardrails are an easy way to enforce system-wide soft and hard
>> limits
>> >> to
>> >>>>> prevent anti-patterns of bad usage and in the long run make it not
>> >>>>> possible
>> >>>>> to severely degrade the performance of a node/cluster through user
>> >>>>> actions
>> >>>>> such as having too many secondary indexes, too large partitions,
>> almost
>> >>>>> full disks, etc.
>> >>>>>
>> >>>>> Thanks,
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>>>>
>> >>
>>
>>
>> -
>> To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org
>> For additional commands, e-mail: dev-h...@cassandra.apache.org
>>
>>
Re: [DISCUSS] CEP-3: Guardrails
om/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java > >>>>> < > >>>>> > >> > https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java > >>> > >>>>> support mutability then we get vtable support. I am cool with JMX > >> and/or > >>>>> vtable, to me its just more important to allow dynamic setting of > these > >>>>> configs. > >>>>> > >>>>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote: > >>>>> > >>>>> having them only configured via yaml seems like a bad outcome > >>>>> > >>>>> > >>>>> +1 > >>>>> > >>>>> I would like to see us move towards configuration being driven > through > >>>>> virtual tables where possible, so that the whole cluster can be > managed > >>>>> from a single interface. Not sure if this is the right place to bite > >> this > >>>>> off, but perhaps? > >>>>> > >>>>> From: Jeff Jirsa > >>>>> Date: Monday, 1 November 2021 at 16:47 > >>>>> To: Cassandra DEV > >>>>> Subject: Re: [DISCUSS] CEP-3: Guardrails > >>>>> Without bike-shedding too much, guardrails would be great, building > >> them > >>>>> into a more general purpose framework that limits various dangerous > >>>>> things > >>>>> would be fantastic. The CEP says that the guardrails should be > distinct > >>>>> from the capability restrictions ( > >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't > >> see > >>>>> why > >>>>> that needs to be the case. A system-level guardrail and a > >> personal-level > >>>>> guardrail are both restrictions, they just have different scopes, so > >>>>> implement the restriction framework first, and allow the scopes to be > >>>>> expanded as needed? > >>>>> > >>>>> Naming wise, I don't know that I'd actually surface these as > >>>>> "guardrails", > >>>>> but more as general "limits", and having them only configured via > yaml > >>>>> seems like a bad outcome > >>>>> > >>>>> > >>>>> > >>>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 > >>>>> > >>>>> > >>>>> > >>>>> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña < > adelap...@apache.org > >>> > >>>>> wrote: > >>>>> > >>>>> Hi everyone, > >>>>> > >>>>> I'd like to start a discussion about Guardrails proposal: > >>>>> > >>>>> > >>>>> > >> > https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails > >>>>> > >>>>> Guardrails are an easy way to enforce system-wide soft and hard > limits > >> to > >>>>> prevent anti-patterns of bad usage and in the long run make it not > >>>>> possible > >>>>> to severely degrade the performance of a node/cluster through user > >>>>> actions > >>>>> such as having too many secondary indexes, too large partitions, > almost > >>>>> full disks, etc. > >>>>> > >>>>> Thanks, > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >>>>> > >> > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > >
Re: [DISCUSS] CEP-3: Guardrails
I am cool with voting. I would love to see added something like “all guard
rails must work in a rolling fashion where some nodes do not have the latest
guard rails”, but also cool leaving this to JIRA.
> On Nov 10, 2021, at 11:17 AM, Ekaterina Dimitrova
> wrote:
>
> I am personally ready to give you my vote :-)
>
> On Wed, 10 Nov 2021 at 13:09, Andrés de la Peña
> wrote:
>
>> I think the updated CEP incorporates the feedback above, unless I'm missing
>> something. Are we ready to start a vote?
>>
>> On Tue, 2 Nov 2021 at 15:54, Andrés de la Peña
>> wrote:
>>
>>> Under "Migrating existing cassandra.yaml warn/fail thresholds”, I
>> recently
>>>> added a few things which are basically guardrails, so should be
>> included in
>>>> this set; they are configured by track_warnings (coordinator_read_size,
>>>> local_read_size, and row_index_size). With track_warnings I setup the
>>>> plumbing to have read queries trigger warnings (or abort the query) to
>> the
>>>> client exists (under "Event logging" you mention "and also to the client
>>>> connection when applicable”) and isn’t limited to the coordinator
>>>> participating in the query (previous limitation for tombstone warnings).
>>>> One thing I found which was problematic for track_warnings was that
>>>> altering clients is annoying as java and python both ignore the error
>>>> message we send (see
>>>>
>> https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131
>> ).
>>>> We log client warnings (if enabled) but ignore any detailed error
>> message
>>>> received from the server; it would be good to talk about client
>>>> integrations and how users are informed of issues in more detail.
>>>
>>>
>>> I have updated the CEP to include those thresholds among the ones that
>>> could be migrated once we have the guardrails framework ready. I have
>> also
>>> mentioned the usage of internal messaging to be able to propagate the
>>> outcome of guardrails triggered on nodes that are not the coordinator,
>> and
>>> the need of making changes on drivers.
>>>
>>> What I meant by "and also to the client connection when applicable" is
>>> that some guardrails can be applied to things that are nor necessarily
>>> associated to a client connection, such as compaction. I have tried to be
>>> more explicit about that.
>>>
>>>
>>> On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña >>
>>> wrote:
>>>
>>>> Being able to configure guardrails dynamically makes a lot of sense to
>>>> me, I have updated the CEP to mention that. I think we don't need to
>> decide
>>>> yet whether it would be done through JMX and/or virtual tables.
>>>>
>>>> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas
>>>> wrote:
>>>>
>>>>> Re: "I think you all know my feels on JMX." –
>>>>>
>>>>> Super fair - I'd meant to speak in terms of desired outcome ("the
>>>>> feature should be dynamically configurable at runtime") rather than
>>>>> implementation ("this should be via JMX"). 👍
>>>>>
>>>>> On Nov 1, 2021, at 1:24 PM, David Capwell
>>>>> wrote:
>>>>>
>>>>>
>>>>> If anyone wants to bite off making
>>>>>
>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
>>>>> <
>>>>>
>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
>>>
>>>>> support mutability then we get vtable support. I am cool with JMX
>> and/or
>>>>> vtable, to me its just more important to allow dynamic setting of these
>>>>> configs.
>>>>>
>>>>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote:
>>>>>
>>>>> having them only configured via yaml seems like a bad outcome
>>>>>
>>>>>
>>>>> +1
>>>>>
>>>>> I would like to see us move towards configuration being driven through
>>>>> virtual tables where possible, so that the whole cluster can
Re: [DISCUSS] CEP-3: Guardrails
I am personally ready to give you my vote :-)
On Wed, 10 Nov 2021 at 13:09, Andrés de la Peña
wrote:
> I think the updated CEP incorporates the feedback above, unless I'm missing
> something. Are we ready to start a vote?
>
> On Tue, 2 Nov 2021 at 15:54, Andrés de la Peña
> wrote:
>
> > Under "Migrating existing cassandra.yaml warn/fail thresholds”, I
> recently
> >> added a few things which are basically guardrails, so should be
> included in
> >> this set; they are configured by track_warnings (coordinator_read_size,
> >> local_read_size, and row_index_size). With track_warnings I setup the
> >> plumbing to have read queries trigger warnings (or abort the query) to
> the
> >> client exists (under "Event logging" you mention "and also to the client
> >> connection when applicable”) and isn’t limited to the coordinator
> >> participating in the query (previous limitation for tombstone warnings).
> >> One thing I found which was problematic for track_warnings was that
> >> altering clients is annoying as java and python both ignore the error
> >> message we send (see
> >>
> https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131
> ).
> >> We log client warnings (if enabled) but ignore any detailed error
> message
> >> received from the server; it would be good to talk about client
> >> integrations and how users are informed of issues in more detail.
> >
> >
> > I have updated the CEP to include those thresholds among the ones that
> > could be migrated once we have the guardrails framework ready. I have
> also
> > mentioned the usage of internal messaging to be able to propagate the
> > outcome of guardrails triggered on nodes that are not the coordinator,
> and
> > the need of making changes on drivers.
> >
> > What I meant by "and also to the client connection when applicable" is
> > that some guardrails can be applied to things that are nor necessarily
> > associated to a client connection, such as compaction. I have tried to be
> > more explicit about that.
> >
> >
> > On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña >
> > wrote:
> >
> >> Being able to configure guardrails dynamically makes a lot of sense to
> >> me, I have updated the CEP to mention that. I think we don't need to
> decide
> >> yet whether it would be done through JMX and/or virtual tables.
> >>
> >> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas
> >> wrote:
> >>
> >>> Re: "I think you all know my feels on JMX." –
> >>>
> >>> Super fair - I'd meant to speak in terms of desired outcome ("the
> >>> feature should be dynamically configurable at runtime") rather than
> >>> implementation ("this should be via JMX"). 👍
> >>>
> >>> On Nov 1, 2021, at 1:24 PM, David Capwell
> >>> wrote:
> >>>
> >>>
> >>> If anyone wants to bite off making
> >>>
> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
> >>> <
> >>>
> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
> >
> >>> support mutability then we get vtable support. I am cool with JMX
> and/or
> >>> vtable, to me its just more important to allow dynamic setting of these
> >>> configs.
> >>>
> >>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote:
> >>>
> >>> having them only configured via yaml seems like a bad outcome
> >>>
> >>>
> >>> +1
> >>>
> >>> I would like to see us move towards configuration being driven through
> >>> virtual tables where possible, so that the whole cluster can be managed
> >>> from a single interface. Not sure if this is the right place to bite
> this
> >>> off, but perhaps?
> >>>
> >>> From: Jeff Jirsa
> >>> Date: Monday, 1 November 2021 at 16:47
> >>> To: Cassandra DEV
> >>> Subject: Re: [DISCUSS] CEP-3: Guardrails
> >>> Without bike-shedding too much, guardrails would be great, building
> them
> >>> into a more general purpose framework that limits various dangerous
> >>> things
> >>> would be fantastic. T
Re: [DISCUSS] CEP-3: Guardrails
I think the updated CEP incorporates the feedback above, unless I'm missing
something. Are we ready to start a vote?
On Tue, 2 Nov 2021 at 15:54, Andrés de la Peña wrote:
> Under "Migrating existing cassandra.yaml warn/fail thresholds”, I recently
>> added a few things which are basically guardrails, so should be included in
>> this set; they are configured by track_warnings (coordinator_read_size,
>> local_read_size, and row_index_size). With track_warnings I setup the
>> plumbing to have read queries trigger warnings (or abort the query) to the
>> client exists (under "Event logging" you mention "and also to the client
>> connection when applicable”) and isn’t limited to the coordinator
>> participating in the query (previous limitation for tombstone warnings).
>> One thing I found which was problematic for track_warnings was that
>> altering clients is annoying as java and python both ignore the error
>> message we send (see
>> https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131).
>> We log client warnings (if enabled) but ignore any detailed error message
>> received from the server; it would be good to talk about client
>> integrations and how users are informed of issues in more detail.
>
>
> I have updated the CEP to include those thresholds among the ones that
> could be migrated once we have the guardrails framework ready. I have also
> mentioned the usage of internal messaging to be able to propagate the
> outcome of guardrails triggered on nodes that are not the coordinator, and
> the need of making changes on drivers.
>
> What I meant by "and also to the client connection when applicable" is
> that some guardrails can be applied to things that are nor necessarily
> associated to a client connection, such as compaction. I have tried to be
> more explicit about that.
>
>
> On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña
> wrote:
>
>> Being able to configure guardrails dynamically makes a lot of sense to
>> me, I have updated the CEP to mention that. I think we don't need to decide
>> yet whether it would be done through JMX and/or virtual tables.
>>
>> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas
>> wrote:
>>
>>> Re: "I think you all know my feels on JMX." –
>>>
>>> Super fair - I'd meant to speak in terms of desired outcome ("the
>>> feature should be dynamically configurable at runtime") rather than
>>> implementation ("this should be via JMX"). 👍
>>>
>>> On Nov 1, 2021, at 1:24 PM, David Capwell
>>> wrote:
>>>
>>>
>>> If anyone wants to bite off making
>>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
>>> <
>>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java>
>>> support mutability then we get vtable support. I am cool with JMX and/or
>>> vtable, to me its just more important to allow dynamic setting of these
>>> configs.
>>>
>>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote:
>>>
>>> having them only configured via yaml seems like a bad outcome
>>>
>>>
>>> +1
>>>
>>> I would like to see us move towards configuration being driven through
>>> virtual tables where possible, so that the whole cluster can be managed
>>> from a single interface. Not sure if this is the right place to bite this
>>> off, but perhaps?
>>>
>>> From: Jeff Jirsa
>>> Date: Monday, 1 November 2021 at 16:47
>>> To: Cassandra DEV
>>> Subject: Re: [DISCUSS] CEP-3: Guardrails
>>> Without bike-shedding too much, guardrails would be great, building them
>>> into a more general purpose framework that limits various dangerous
>>> things
>>> would be fantastic. The CEP says that the guardrails should be distinct
>>> from the capability restrictions (
>>> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see
>>> why
>>> that needs to be the case. A system-level guardrail and a personal-level
>>> guardrail are both restrictions, they just have different scopes, so
>>> implement the restriction framework first, and allow the scopes to be
>>> expanded as needed?
>>>
>>> Naming wise, I don't know that I'd actually surface these as
>>> "guar
Re: [DISCUSS] CEP-3: Guardrails
>
> Under "Migrating existing cassandra.yaml warn/fail thresholds”, I recently
> added a few things which are basically guardrails, so should be included in
> this set; they are configured by track_warnings (coordinator_read_size,
> local_read_size, and row_index_size). With track_warnings I setup the
> plumbing to have read queries trigger warnings (or abort the query) to the
> client exists (under "Event logging" you mention "and also to the client
> connection when applicable”) and isn’t limited to the coordinator
> participating in the query (previous limitation for tombstone warnings).
> One thing I found which was problematic for track_warnings was that
> altering clients is annoying as java and python both ignore the error
> message we send (see
> https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131).
> We log client warnings (if enabled) but ignore any detailed error message
> received from the server; it would be good to talk about client
> integrations and how users are informed of issues in more detail.
I have updated the CEP to include those thresholds among the ones that
could be migrated once we have the guardrails framework ready. I have also
mentioned the usage of internal messaging to be able to propagate the
outcome of guardrails triggered on nodes that are not the coordinator, and
the need of making changes on drivers.
What I meant by "and also to the client connection when applicable" is that
some guardrails can be applied to things that are nor necessarily
associated to a client connection, such as compaction. I have tried to be
more explicit about that.
On Tue, 2 Nov 2021 at 12:53, Andrés de la Peña
wrote:
> Being able to configure guardrails dynamically makes a lot of sense to me,
> I have updated the CEP to mention that. I think we don't need to decide yet
> whether it would be done through JMX and/or virtual tables.
>
> On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas
> wrote:
>
>> Re: "I think you all know my feels on JMX." –
>>
>> Super fair - I'd meant to speak in terms of desired outcome ("the feature
>> should be dynamically configurable at runtime") rather than implementation
>> ("this should be via JMX"). 👍
>>
>> On Nov 1, 2021, at 1:24 PM, David Capwell
>> wrote:
>>
>>
>> If anyone wants to bite off making
>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
>> <
>> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java>
>> support mutability then we get vtable support. I am cool with JMX and/or
>> vtable, to me its just more important to allow dynamic setting of these
>> configs.
>>
>> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote:
>>
>> having them only configured via yaml seems like a bad outcome
>>
>>
>> +1
>>
>> I would like to see us move towards configuration being driven through
>> virtual tables where possible, so that the whole cluster can be managed
>> from a single interface. Not sure if this is the right place to bite this
>> off, but perhaps?
>>
>> From: Jeff Jirsa
>> Date: Monday, 1 November 2021 at 16:47
>> To: Cassandra DEV
>> Subject: Re: [DISCUSS] CEP-3: Guardrails
>> Without bike-shedding too much, guardrails would be great, building them
>> into a more general purpose framework that limits various dangerous things
>> would be fantastic. The CEP says that the guardrails should be distinct
>> from the capability restrictions (
>> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see
>> why
>> that needs to be the case. A system-level guardrail and a personal-level
>> guardrail are both restrictions, they just have different scopes, so
>> implement the restriction framework first, and allow the scopes to be
>> expanded as needed?
>>
>> Naming wise, I don't know that I'd actually surface these as "guardrails",
>> but more as general "limits", and having them only configured via yaml
>> seems like a bad outcome
>>
>>
>>
>> https://issues.apache.org/jira/browse/CASSANDRA-8303
>>
>>
>>
>> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña
>> wrote:
>>
>> Hi everyone,
>>
>> I'd like to start a discussion about Guardrails proposal:
>>
>>
>> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>>
>> Guardrails are an easy way to enforce system-wide soft and hard limits to
>> prevent anti-patterns of bad usage and in the long run make it not
>> possible
>> to severely degrade the performance of a node/cluster through user actions
>> such as having too many secondary indexes, too large partitions, almost
>> full disks, etc.
>>
>> Thanks,
>>
>>
>>
>>
>>
>>
Re: [DISCUSS] CEP-3: Guardrails
Being able to configure guardrails dynamically makes a lot of sense to me,
I have updated the CEP to mention that. I think we don't need to decide yet
whether it would be done through JMX and/or virtual tables.
On Mon, 1 Nov 2021 at 20:35, C. Scott Andreas wrote:
> Re: "I think you all know my feels on JMX." –
>
> Super fair - I'd meant to speak in terms of desired outcome ("the feature
> should be dynamically configurable at runtime") rather than implementation
> ("this should be via JMX"). 👍
>
> On Nov 1, 2021, at 1:24 PM, David Capwell
> wrote:
>
>
> If anyone wants to bite off making
> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
> <
> https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java>
> support mutability then we get vtable support. I am cool with JMX and/or
> vtable, to me its just more important to allow dynamic setting of these
> configs.
>
> On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote:
>
> having them only configured via yaml seems like a bad outcome
>
>
> +1
>
> I would like to see us move towards configuration being driven through
> virtual tables where possible, so that the whole cluster can be managed
> from a single interface. Not sure if this is the right place to bite this
> off, but perhaps?
>
> From: Jeff Jirsa
> Date: Monday, 1 November 2021 at 16:47
> To: Cassandra DEV
> Subject: Re: [DISCUSS] CEP-3: Guardrails
> Without bike-shedding too much, guardrails would be great, building them
> into a more general purpose framework that limits various dangerous things
> would be fantastic. The CEP says that the guardrails should be distinct
> from the capability restrictions (
> https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see
> why
> that needs to be the case. A system-level guardrail and a personal-level
> guardrail are both restrictions, they just have different scopes, so
> implement the restriction framework first, and allow the scopes to be
> expanded as needed?
>
> Naming wise, I don't know that I'd actually surface these as "guardrails",
> but more as general "limits", and having them only configured via yaml
> seems like a bad outcome
>
>
>
> https://issues.apache.org/jira/browse/CASSANDRA-8303
>
>
>
> On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña
> wrote:
>
> Hi everyone,
>
> I'd like to start a discussion about Guardrails proposal:
>
>
> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails
>
> Guardrails are an easy way to enforce system-wide soft and hard limits to
> prevent anti-patterns of bad usage and in the long run make it not possible
> to severely degrade the performance of a node/cluster through user actions
> such as having too many secondary indexes, too large partitions, almost
> full disks, etc.
>
> Thanks,
>
>
>
>
>
>
Re: [DISCUSS] CEP-3: Guardrails
Re: "I think you all know my feels on JMX." –Super fair - I'd meant to speak in terms of desired outcome ("the feature should be dynamically
configurable at runtime") rather than implementation ("this should be via JMX"). 👍On Nov 1, 2021, at 1:24 PM, David Capwell
wrote:If anyone wants to bite off making
https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java
support
mutability then we get vtable support. I am cool with JMX and/or vtable, to me its just more important to allow dynamic setting of these configs.On Nov 1,
2021, at 10:36 AM, bened...@apache.org wrote:having them only configured via yaml seems like a bad outcome+1I would like to see us move towards configuration
being driven through virtual tables where possible, so that the whole cluster can be managed from a single interface. Not sure if this is the right place to
bite this off, but perhaps?From: Jeff Jirsa Date: Monday, 1 November 2021 at 16:47To: Cassandra DEV
Subject: Re: [DISCUSS] CEP-3: GuardrailsWithout bike-shedding too much, guardrails would be great, building theminto a more
general purpose framework that limits various dangerous thingswould be fantastic. The CEP says that the guardrails should be distinctfrom the capability
restrictions (https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see whythat needs to be the case. A system-level guardrail and a
personal-levelguardrail are both restrictions, they just have different scopes, soimplement the restriction framework first, and allow the scopes to beexpanded
as needed?Naming wise, I don't know that I'd actually surface these as "guardrails",but more as general "limits", and having them only
configured via yamlseems like a bad outcomehttps://issues.apache.org/jira/browse/CASSANDRA-8303On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña
wrote:Hi everyone,I'd like to start a discussion about Guardrails
proposal:https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+GuardrailsGuardrails are an easy way to enforce system-wide soft and hard
limits toprevent anti-patterns of bad usage and in the long run make it not possibleto severely degrade the performance of a node/cluster through user
actionssuch as having too many secondary indexes, too large partitions, almostfull disks, etc.Thanks,
Re: [DISCUSS] CEP-3: Guardrails
If anyone wants to bite off making https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java <https://github.com/apache/cassandra/blob/ab920c30310a8c095ba76b363142b8e74cbf0a0a/src/java/org/apache/cassandra/db/virtual/SettingsTable.java> support mutability then we get vtable support. I am cool with JMX and/or vtable, to me its just more important to allow dynamic setting of these configs. > On Nov 1, 2021, at 10:36 AM, bened...@apache.org wrote: > >> having them only configured via yaml seems like a bad outcome > > +1 > > I would like to see us move towards configuration being driven through > virtual tables where possible, so that the whole cluster can be managed from > a single interface. Not sure if this is the right place to bite this off, but > perhaps? > > From: Jeff Jirsa > Date: Monday, 1 November 2021 at 16:47 > To: Cassandra DEV > Subject: Re: [DISCUSS] CEP-3: Guardrails > Without bike-shedding too much, guardrails would be great, building them > into a more general purpose framework that limits various dangerous things > would be fantastic. The CEP says that the guardrails should be distinct > from the capability restrictions ( > https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why > that needs to be the case. A system-level guardrail and a personal-level > guardrail are both restrictions, they just have different scopes, so > implement the restriction framework first, and allow the scopes to be > expanded as needed? > > Naming wise, I don't know that I'd actually surface these as "guardrails", > but more as general "limits", and having them only configured via yaml > seems like a bad outcome > > > > https://issues.apache.org/jira/browse/CASSANDRA-8303 > > > > On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña > wrote: > >> Hi everyone, >> >> I'd like to start a discussion about Guardrails proposal: >> >> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails >> >> Guardrails are an easy way to enforce system-wide soft and hard limits to >> prevent anti-patterns of bad usage and in the long run make it not possible >> to severely degrade the performance of a node/cluster through user actions >> such as having too many secondary indexes, too large partitions, almost >> full disks, etc. >> >> Thanks, >>
Re: [DISCUSS] CEP-3: Guardrails
> having them only configured via yaml seems like a bad outcome +1 I would like to see us move towards configuration being driven through virtual tables where possible, so that the whole cluster can be managed from a single interface. Not sure if this is the right place to bite this off, but perhaps? From: Jeff Jirsa Date: Monday, 1 November 2021 at 16:47 To: Cassandra DEV Subject: Re: [DISCUSS] CEP-3: Guardrails Without bike-shedding too much, guardrails would be great, building them into a more general purpose framework that limits various dangerous things would be fantastic. The CEP says that the guardrails should be distinct from the capability restrictions ( https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why that needs to be the case. A system-level guardrail and a personal-level guardrail are both restrictions, they just have different scopes, so implement the restriction framework first, and allow the scopes to be expanded as needed? Naming wise, I don't know that I'd actually surface these as "guardrails", but more as general "limits", and having them only configured via yaml seems like a bad outcome https://issues.apache.org/jira/browse/CASSANDRA-8303 On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña wrote: > Hi everyone, > > I'd like to start a discussion about Guardrails proposal: > > https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails > > Guardrails are an easy way to enforce system-wide soft and hard limits to > prevent anti-patterns of bad usage and in the long run make it not possible > to severely degrade the performance of a node/cluster through user actions > such as having too many secondary indexes, too large partitions, almost > full disks, etc. > > Thanks, >
Re: [DISCUSS] CEP-3: Guardrails
"it will be important that these guardrails can be modified via JMX as well" I think you all know my feels on JMX. Maybe this is something we can go straight to virtual tables? On Mon, Nov 1, 2021 at 12:12 PM C. Scott Andreas wrote: > Thank you for starting discussion on this CEP, Andrés! > > Can the "Scope" section of the doc be filled out? It currently reads > "TBD," but having a better understanding of the scope of work would help > focus discussion: > https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-3%3A+Guardrails > > Re: configuration via yaml, it will be important that these guardrails can > be modified via JMX as well - e.g., in the case of a user running up > against a limit and needing a path to being unblocked that doesn't require > a yaml change and rolling restart. > > As David mentions, raising the visibility of the soft limit warnings will > help users avoid being caught off-guard. Enabling the logging of wire > protocol warnings received in CQL responses in the drivers by default would > help if related JIRA tickets in those projects can be considered. > > On Nov 1, 2021, at 10:05 AM, David Capwell > wrote: > > > Under "Migrating existing cassandra.yaml warn/fail thresholds”, I recently > added a few things which are basically guardrails, so should be included in > this set; they are configured by track_warnings (coordinator_read_size, > local_read_size, and row_index_size). With track_warnings I setup the > plumbing to have read queries trigger warnings (or abort the query) to the > client exists (under "Event logging" you mention "and also to the client > connection when applicable”) and isn’t limited to the coordinator > participating in the query (previous limitation for tombstone warnings). > One thing I found which was problematic for track_warnings was that > altering clients is annoying as java and python both ignore the error > message we send (see > https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131). > We log client warnings (if enabled) but ignore any detailed error message > received from the server; it would be good to talk about client > integrations and how users are informed of issues in more detail. > > > On Nov 1, 2021, at 9:46 AM, Jeff Jirsa wrote: > > Without bike-shedding too much, guardrails would be great, building them > into a more general purpose framework that limits various dangerous things > would be fantastic. The CEP says that the guardrails should be distinct > from the capability restrictions ( > https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see > why > that needs to be the case. A system-level guardrail and a personal-level > guardrail are both restrictions, they just have different scopes, so > implement the restriction framework first, and allow the scopes to be > expanded as needed? > > Naming wise, I don't know that I'd actually surface these as "guardrails", > but more as general "limits", and having them only configured via yaml > seems like a bad outcome > > > > https://issues.apache.org/jira/browse/CASSANDRA-8303 > > > > On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña > wrote: > > Hi everyone, > > I'd like to start a discussion about Guardrails proposal: > > > https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails > > Guardrails are an easy way to enforce system-wide soft and hard limits to > prevent anti-patterns of bad usage and in the long run make it not possible > to severely degrade the performance of a node/cluster through user actions > such as having too many secondary indexes, too large partitions, almost > full disks, etc. > > Thanks, > > > > - > To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org > For additional commands, e-mail: dev-h...@cassandra.apache.org > > > > >
Re: [DISCUSS] CEP-3: Guardrails
Thank you for starting discussion on this CEP, Andrés!Can the "Scope" section of the doc be filled out? It currently reads "TBD," but having a better understanding of the scope of work would help focus discussion: https://cwiki.apache.org/confluence/display/CASSANDRA/CEP-3%3A+GuardrailsRe: configuration via yaml, it will be important that these guardrails can be modified via JMX as well - e.g., in the case of a user running up against a limit and needing a path to being unblocked that doesn't require a yaml change and rolling restart.As David mentions, raising the visibility of the soft limit warnings will help users avoid being caught off-guard. Enabling the logging of wire protocol warnings received in CQL responses in the drivers by default would help if related JIRA tickets in those projects can be considered.On Nov 1, 2021, at 10:05 AM, David Capwell wrote:Under "Migrating existing cassandra.yaml warn/fail thresholds”, I recently added a few things which are basically guardrails, so should be included in this set; they are configured by track_warnings (coordinator_read_size, local_read_size, and row_index_size). With track_warnings I setup the plumbing to have read queries trigger warnings (or abort the query) to the client exists (under "Event logging" you mention "and also to the client connection when applicable”) and isn’t limited to the coordinator participating in the query (previous limitation for tombstone warnings). One thing I found which was problematic for track_warnings was that altering clients is annoying as java and python both ignore the error message we send (see https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131). We log client warnings (if enabled) but ignore any detailed error message received from the server; it would be good to talk about client integrations and how users are informed of issues in more detail.On Nov 1, 2021, at 9:46 AM, Jeff Jirsa wrote:Without bike-shedding too much, guardrails would be great, building theminto a more general purpose framework that limits various dangerous thingswould be fantastic. The CEP says that the guardrails should be distinctfrom the capability restrictions (https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see whythat needs to be the case. A system-level guardrail and a personal-levelguardrail are both restrictions, they just have different scopes, soimplement the restriction framework first, and allow the scopes to beexpanded as needed?Naming wise, I don't know that I'd actually surface these as "guardrails",but more as general "limits", and having them only configured via yamlseems like a bad outcomehttps://issues.apache.org/jira/browse/CASSANDRA-8303On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña wrote:Hi everyone,I'd like to start a discussion about Guardrails proposal:https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+GuardrailsGuardrails are an easy way to enforce system-wide soft and hard limits toprevent anti-patterns of bad usage and in the long run make it not possibleto severely degrade the performance of a node/cluster through user actionssuch as having too many secondary indexes, too large partitions, almostfull disks, etc.Thanks,-To unsubscribe, e-mail: dev-unsubscribe@cassandra.apache.orgFor additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CEP-3: Guardrails
Under "Migrating existing cassandra.yaml warn/fail thresholds”, I recently added a few things which are basically guardrails, so should be included in this set; they are configured by track_warnings (coordinator_read_size, local_read_size, and row_index_size). With track_warnings I setup the plumbing to have read queries trigger warnings (or abort the query) to the client exists (under "Event logging" you mention "and also to the client connection when applicable”) and isn’t limited to the coordinator participating in the query (previous limitation for tombstone warnings). One thing I found which was problematic for track_warnings was that altering clients is annoying as java and python both ignore the error message we send (see https://github.com/datastax/java-driver/blob/3.11.0/driver-core/src/main/java/com/datastax/driver/core/Responses.java#L73-L131). We log client warnings (if enabled) but ignore any detailed error message received from the server; it would be good to talk about client integrations and how users are informed of issues in more detail. > On Nov 1, 2021, at 9:46 AM, Jeff Jirsa wrote: > > Without bike-shedding too much, guardrails would be great, building them > into a more general purpose framework that limits various dangerous things > would be fantastic. The CEP says that the guardrails should be distinct > from the capability restrictions ( > https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why > that needs to be the case. A system-level guardrail and a personal-level > guardrail are both restrictions, they just have different scopes, so > implement the restriction framework first, and allow the scopes to be > expanded as needed? > > Naming wise, I don't know that I'd actually surface these as "guardrails", > but more as general "limits", and having them only configured via yaml > seems like a bad outcome > > > > https://issues.apache.org/jira/browse/CASSANDRA-8303 > > > > On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña > wrote: > >> Hi everyone, >> >> I'd like to start a discussion about Guardrails proposal: >> >> https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails >> >> Guardrails are an easy way to enforce system-wide soft and hard limits to >> prevent anti-patterns of bad usage and in the long run make it not possible >> to severely degrade the performance of a node/cluster through user actions >> such as having too many secondary indexes, too large partitions, almost >> full disks, etc. >> >> Thanks, >> - To unsubscribe, e-mail: dev-unsubscr...@cassandra.apache.org For additional commands, e-mail: dev-h...@cassandra.apache.org
Re: [DISCUSS] CEP-3: Guardrails
Without bike-shedding too much, guardrails would be great, building them into a more general purpose framework that limits various dangerous things would be fantastic. The CEP says that the guardrails should be distinct from the capability restrictions ( https://issues.apache.org/jira/browse/CASSANDRA-8303 ), but I don't see why that needs to be the case. A system-level guardrail and a personal-level guardrail are both restrictions, they just have different scopes, so implement the restriction framework first, and allow the scopes to be expanded as needed? Naming wise, I don't know that I'd actually surface these as "guardrails", but more as general "limits", and having them only configured via yaml seems like a bad outcome https://issues.apache.org/jira/browse/CASSANDRA-8303 On Mon, Nov 1, 2021 at 9:31 AM Andrés de la Peña wrote: > Hi everyone, > > I'd like to start a discussion about Guardrails proposal: > > https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails > > Guardrails are an easy way to enforce system-wide soft and hard limits to > prevent anti-patterns of bad usage and in the long run make it not possible > to severely degrade the performance of a node/cluster through user actions > such as having too many secondary indexes, too large partitions, almost > full disks, etc. > > Thanks, >
[DISCUSS] CEP-3: Guardrails
Hi everyone, I'd like to start a discussion about Guardrails proposal: https://cwiki.apache.org/confluence/display/CASSANDRA/%28DRAFT%29+-+CEP-3%3A+Guardrails Guardrails are an easy way to enforce system-wide soft and hard limits to prevent anti-patterns of bad usage and in the long run make it not possible to severely degrade the performance of a node/cluster through user actions such as having too many secondary indexes, too large partitions, almost full disks, etc. Thanks,
