Re: [DISCUSS] Update default disk_access_mode to mmap_index_only on 5.0

[Caleb Rackliffe]

Added one nit to the PR. Otherwise, this is awesome :)

On Wed, Nov 15, 2023 at 11:01 AM Jordan West  wrote:

> I would also like to back this proposal. We change this default because
> several incidents have occurred by leaving the default of auto. There are
> rare cases where auto/mmap is the better option but as for a default
> mmap_index_only is safer.
>
> On Wed, Nov 15, 2023 at 6:35 AM Paulo Motta  wrote:
>
>> Hi,
>>
>> I would like to get back to this. I proposed this default configuration
>> change on the user list ~1 month ago and there were no comments [1].
>>
>> I created CASSANDRA-19021 [2] to make the proposed change and Stefan
>> kindly submitted a patch, CI is looking good.
>>
>> Any objections to making this change in 5.0? If not, we will merge in 24
>> hours.
>>
>> Thanks,
>>
>> Paulo
>>
>> [1] - https://lists.apache.org/thread/w0gkdj7fhylycqwmd73p0kfck7jr8qth
>> [2] - https://issues.apache.org/jira/browse/CASSANDRA-19021
>>
>> On Wed, Sep 6, 2023 at 5:12 PM Paulo Motta 
>> wrote:
>>
>>> > I wonder why disk_access_mode property is not in cassandra.yaml
>>> (looking into trunk right now)
>>>
>>> I think there's a prehistoric reason why it was removed but I can't
>>> remember right now.
>>>
>>> > Do you all think we can add it there with brief explanation what each
>>> option does?
>>>
>>> We could reinclude it as long as we provide a clear recommendation on
>>> when to change from the default since this is an advanced setting which
>>> should be rarely changed. But I still think we should provide a more
>>> stable/foolproof default (mmap_index_only) since the current default (mmap)
>>> is known to cause instability in some scenarios.
>>>
>>> Also there is a technicality with changing the default, if we change the
>>> "auto" behavior from mmap to mmap_index_only this may affect users relying
>>> on the default "mmap" behavior. Not sure the best way to address that, is a
>>> big NEWS note sufficient? Even though users are expected to read NEWS when
>>> upgrading we know well not all users read it.
>>>
>>> > Shall we also share this thread with @user?
>>>
>>> Thanks Ekaterina! If we decide to change the default we can run this
>>> through the user@ list to see what the user community thinks.
>>>
>>> On Wed, Sep 6, 2023 at 4:45 PM Ekaterina Dimitrova <
>>> e.dimitr...@gmail.com> wrote:
>>>
 Thanks for starting this discussion, Paulo!

 Shall we also share this thread with @user?

 On Wed, 6 Sep 2023 at 16:35, C. Scott Andreas 
 wrote:

> Supportive of switching the default to mmap_index_only as well.
>
> I don’t have numbers handy to share, but my experience has been
> significantly lower read latency and I wouldn’t run with auto. I’ve also
> not observed substantial heap pressure after switching - it was strictly 
> an
> improvement.
>
> - Scott
>
> —
> Mobile
>
> On Sep 6, 2023, at 8:50 AM, Paulo Motta 
> wrote:
>
> 
>
> Hi,
>
> I've been bitten by OOMs with disk_access_mode:auto/mmap that were
> fixed by changing to disk_access_mode:mmap_index_only. In a particular
> benchmark I got 5x more read throughput on 3.11.x with disk_access_mode:
> mmap_index_only vs disk_access_mode: auto/mmap.
>
> Changing disk_access_mode to mmap_index_only seems to be a common
> recommendation on forums[1][2][3][4] and slack (find by searching
> disk_access_mode in the #cassandra channel on
> https://the-asf.slack.com/).
>
> It's not clear to me when using the default
> disk_access_mode:auto/mmap is beneficial, perhaps only when the read set
> fits in memory? Mick seems to think on CASSANDRA-15531 [5], that
> mmap_index_only has a higher heap cost and should be only used when
> warranted. However it's not uncommon to see people being bitten with OOMs
> or lower read performance due to the default disk_access_mode, so it makes
> me think it's not the best fool-proof default.
>
> Should we consider changing default "auto" behavior of
> "disk_access_mode" to be "mmap_index_only" instead of "mmap" in 5.0 since
> it's likely safer and perhaps more performant?
>
> Thanks,
>
> Paulo
>
> [1]
> https://stackoverflow.com/questions/72272035/troubleshooting-and-fixing-cassandra-oom-issue
> [2] https://phabricator.wikimedia.org/T137419
> [3] https://stackoverflow.com/a/55975471
> [4]
> https://support.datastax.com/s/article/FAQ-Use-of-disk-access-mode-in-DSE-51-and-earlier
> [5] https://issues.apache.org/jira/browse/CASSANDRA-15531
>
>

Re: [DISCUSSION] CEP-38: CQL Management API

[Maxim Muzafarov]

Hello German,

Thanks for the links, I've seen this project before, but to be honest
I've never delved that deep into it. I'll definitely check it out for
more details, give me a few days to be in context!

As for the admin port, it's already part of the proposal, as discussed
in Slack. This port is needed not only because we are mixing the data
and control planes, but also because the native protocol can be
disabled, e.g. manually via the nodetool disablebinary command, or via
the disk_failure_policy 'stop' policy, which shuts down all
transports, leaving a node only operable via JMX which doesn't match
our goals.



On Wed, 15 Nov 2023 at 18:52, German Eichberger via dev
 wrote:
>
> Hi Maxim,
>
> We have adopted/forked the agent part of the 
> https://github.com/k8ssandra/management-api-for-apache-cassandra project 
> which aims to do similar things. I especially like how they have a local 
> database socket where a sidecar can easily access cassandra and execute cql 
> commands without the need of a service account like your example suggests.
>
> The syntax they adopted (see for instance 
> https://github.com/k8ssandra/management-api-for-apache-cassandra/blob/7cb367eac46a12947bb87486456d3f905f37628b/management-api-server/src/main/java/com/datastax/mgmtapi/resources/NodeOpsResources.java#L115)
>  looks like `CALL NodeOps.decommission(?, ?)", force, false)` which is 
> similar to your execute - just throwing this out as another example.
>
> I definitely like settling on the cql interface since that avoids having to 
> load different jmx bindings for different Cassandra versions making things 
> cleaner and more easily accessible. There is some security concern to mix 
> data and control plane so I would liek to see some way to restrict access 
> like the mgmt api does where the admin commands are only available on the 
> socket. Maybe, have a special admin port or socket?
>
> I  prefer making the agent part of the managment api become part of Cassandra 
> either through your CEP or other means but I can also see this as an adjacent 
> sub project  - let's discuss 
>
> German
>
> 
> From: Maxim Muzafarov 
> Sent: Monday, November 13, 2023 10:08 AM
> To: dev@cassandra.apache.org 
> Subject: [EXTERNAL] [DISCUSSION] CEP-38: CQL Management API
>
> Hello everyone,
>
> While we are still waiting for the review to make the settings virtual
> table updatable (CASSANDRA-15254), which will improve the
> configuration management experience for users, I'd like to take
> another step forward and improve the C* management approach we have as
> a whole. This approach aims to make all Cassandra management commands
> accessible via CQL, but not only that.
>
> The problem of making commands accessible via CQL presents a complex
> challenge, especially if we aim to minimize code duplication across
> the implementation of management operations for different APIs and
> reduce the overall maintenance burden. The proposal's scope goes
> beyond simply introducing a new CQL syntax. It encompasses several key
> objectives for C* management operations, beyond their availability
> through CQL:
> - Ensure consistency across all public APIs we support, including JMX
> MBeans and the newly introduced CQL. Users should see consistent
> command specifications and arguments, irrespective of whether they're
> using an API or a CLI;
> - Reduce source code maintenance costs. With this new approach, when a
> new command is implemented, it should automatically become available
> across JMX MBeans, nodetool, CQL, and Cassandra Sidecar, eliminating
> the need for additional coding;
> - Maintain backward compatibility, ensuring that existing setups and
> workflows continue to work the same way as they do today;
>
> I would suggest discussing the overall design concept first, and then
> diving into the CQL command syntax and other details once we've found
> common ground on the community's vision. However, regardless of these
> details, I would appreciate any feedback on the design.
>
> I look forward to your comments!
>
> Please, see the design document: CEP-38: CQL Management API
> https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCASSANDRA%2FCEP-38%253A%2BCQL%2BManagement%2BAPI=05%7C01%7CGerman.Eichberger%40microsoft.com%7C62051e1eb8964889962d08dbe473d482%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638354958369996874%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=XT4LB1CopZy8qCUM6MnUfBhGFwKHmsUO%2B2AUpgv83zI%3D=0

Re: [DISCUSSION] CEP-38: CQL Management API

[German Eichberger via dev]

Hi Maxim,

We have adopted/forked the agent part of the 
https://github.com/k8ssandra/management-api-for-apache-cassandra project which 
aims to do similar things. I especially like how they have a local database 
socket where a sidecar can easily access cassandra and execute cql commands 
without the need of a service account like your example suggests.

The syntax they adopted (see for instance 
https://github.com/k8ssandra/management-api-for-apache-cassandra/blob/7cb367eac46a12947bb87486456d3f905f37628b/management-api-server/src/main/java/com/datastax/mgmtapi/resources/NodeOpsResources.java#L115)
 looks like `CALL NodeOps.decommission(?, ?)", force, false)` which is similar 
to your execute - just throwing this out as another example.

I definitely like settling on the cql interface since that avoids having to 
load different jmx bindings for different Cassandra versions making things 
cleaner and more easily accessible. There is some security concern to mix data 
and control plane so I would liek to see some way to restrict access like the 
mgmt api does where the admin commands are only available on the socket. Maybe, 
have a special admin port or socket?

I  prefer making the agent part of the managment api become part of Cassandra 
either through your CEP or other means but I can also see this as an adjacent 
sub project  - let's discuss 

German


From: Maxim Muzafarov 
Sent: Monday, November 13, 2023 10:08 AM
To: dev@cassandra.apache.org 
Subject: [EXTERNAL] [DISCUSSION] CEP-38: CQL Management API

Hello everyone,

While we are still waiting for the review to make the settings virtual
table updatable (CASSANDRA-15254), which will improve the
configuration management experience for users, I'd like to take
another step forward and improve the C* management approach we have as
a whole. This approach aims to make all Cassandra management commands
accessible via CQL, but not only that.

The problem of making commands accessible via CQL presents a complex
challenge, especially if we aim to minimize code duplication across
the implementation of management operations for different APIs and
reduce the overall maintenance burden. The proposal's scope goes
beyond simply introducing a new CQL syntax. It encompasses several key
objectives for C* management operations, beyond their availability
through CQL:
- Ensure consistency across all public APIs we support, including JMX
MBeans and the newly introduced CQL. Users should see consistent
command specifications and arguments, irrespective of whether they're
using an API or a CLI;
- Reduce source code maintenance costs. With this new approach, when a
new command is implemented, it should automatically become available
across JMX MBeans, nodetool, CQL, and Cassandra Sidecar, eliminating
the need for additional coding;
- Maintain backward compatibility, ensuring that existing setups and
workflows continue to work the same way as they do today;

I would suggest discussing the overall design concept first, and then
diving into the CQL command syntax and other details once we've found
common ground on the community's vision. However, regardless of these
details, I would appreciate any feedback on the design.

I look forward to your comments!

Please, see the design document: CEP-38: CQL Management API
https://nam06.safelinks.protection.outlook.com/?url=https%3A%2F%2Fcwiki.apache.org%2Fconfluence%2Fdisplay%2FCASSANDRA%2FCEP-38%253A%2BCQL%2BManagement%2BAPI=05%7C01%7CGerman.Eichberger%40microsoft.com%7C62051e1eb8964889962d08dbe473d482%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C638354958369996874%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C3000%7C%7C%7C=XT4LB1CopZy8qCUM6MnUfBhGFwKHmsUO%2B2AUpgv83zI%3D=0

Re: [DISCUSS] Update default disk_access_mode to mmap_index_only on 5.0

[Jordan West]

I would also like to back this proposal. We change this default because
several incidents have occurred by leaving the default of auto. There are
rare cases where auto/mmap is the better option but as for a default
mmap_index_only is safer.

On Wed, Nov 15, 2023 at 6:35 AM Paulo Motta  wrote:

> Hi,
>
> I would like to get back to this. I proposed this default configuration
> change on the user list ~1 month ago and there were no comments [1].
>
> I created CASSANDRA-19021 [2] to make the proposed change and Stefan
> kindly submitted a patch, CI is looking good.
>
> Any objections to making this change in 5.0? If not, we will merge in 24
> hours.
>
> Thanks,
>
> Paulo
>
> [1] - https://lists.apache.org/thread/w0gkdj7fhylycqwmd73p0kfck7jr8qth
> [2] - https://issues.apache.org/jira/browse/CASSANDRA-19021
>
> On Wed, Sep 6, 2023 at 5:12 PM Paulo Motta 
> wrote:
>
>> > I wonder why disk_access_mode property is not in cassandra.yaml
>> (looking into trunk right now)
>>
>> I think there's a prehistoric reason why it was removed but I can't
>> remember right now.
>>
>> > Do you all think we can add it there with brief explanation what each
>> option does?
>>
>> We could reinclude it as long as we provide a clear recommendation on
>> when to change from the default since this is an advanced setting which
>> should be rarely changed. But I still think we should provide a more
>> stable/foolproof default (mmap_index_only) since the current default (mmap)
>> is known to cause instability in some scenarios.
>>
>> Also there is a technicality with changing the default, if we change the
>> "auto" behavior from mmap to mmap_index_only this may affect users relying
>> on the default "mmap" behavior. Not sure the best way to address that, is a
>> big NEWS note sufficient? Even though users are expected to read NEWS when
>> upgrading we know well not all users read it.
>>
>> > Shall we also share this thread with @user?
>>
>> Thanks Ekaterina! If we decide to change the default we can run this
>> through the user@ list to see what the user community thinks.
>>
>> On Wed, Sep 6, 2023 at 4:45 PM Ekaterina Dimitrova 
>> wrote:
>>
>>> Thanks for starting this discussion, Paulo!
>>>
>>> Shall we also share this thread with @user?
>>>
>>> On Wed, 6 Sep 2023 at 16:35, C. Scott Andreas 
>>> wrote:
>>>
 Supportive of switching the default to mmap_index_only as well.

 I don’t have numbers handy to share, but my experience has been
 significantly lower read latency and I wouldn’t run with auto. I’ve also
 not observed substantial heap pressure after switching - it was strictly an
 improvement.

 - Scott

 —
 Mobile

 On Sep 6, 2023, at 8:50 AM, Paulo Motta 
 wrote:

 

 Hi,

 I've been bitten by OOMs with disk_access_mode:auto/mmap that were
 fixed by changing to disk_access_mode:mmap_index_only. In a particular
 benchmark I got 5x more read throughput on 3.11.x with disk_access_mode:
 mmap_index_only vs disk_access_mode: auto/mmap.

 Changing disk_access_mode to mmap_index_only seems to be a common
 recommendation on forums[1][2][3][4] and slack (find by searching
 disk_access_mode in the #cassandra channel on
 https://the-asf.slack.com/).

 It's not clear to me when using the default
 disk_access_mode:auto/mmap is beneficial, perhaps only when the read set
 fits in memory? Mick seems to think on CASSANDRA-15531 [5], that
 mmap_index_only has a higher heap cost and should be only used when
 warranted. However it's not uncommon to see people being bitten with OOMs
 or lower read performance due to the default disk_access_mode, so it makes
 me think it's not the best fool-proof default.

 Should we consider changing default "auto" behavior of
 "disk_access_mode" to be "mmap_index_only" instead of "mmap" in 5.0 since
 it's likely safer and perhaps more performant?

 Thanks,

 Paulo

 [1]
 https://stackoverflow.com/questions/72272035/troubleshooting-and-fixing-cassandra-oom-issue
 [2] https://phabricator.wikimedia.org/T137419
 [3] https://stackoverflow.com/a/55975471
 [4]
 https://support.datastax.com/s/article/FAQ-Use-of-disk-access-mode-in-DSE-51-and-earlier
 [5] https://issues.apache.org/jira/browse/CASSANDRA-15531

Re: [DISCUSS] Update default disk_access_mode to mmap_index_only on 5.0

[Paulo Motta]

Hi,

I would like to get back to this. I proposed this default configuration
change on the user list ~1 month ago and there were no comments [1].

I created CASSANDRA-19021 [2] to make the proposed change and Stefan kindly
submitted a patch, CI is looking good.

Any objections to making this change in 5.0? If not, we will merge in 24
hours.

Thanks,

Paulo

[1] - https://lists.apache.org/thread/w0gkdj7fhylycqwmd73p0kfck7jr8qth
[2] - https://issues.apache.org/jira/browse/CASSANDRA-19021

On Wed, Sep 6, 2023 at 5:12 PM Paulo Motta  wrote:

> > I wonder why disk_access_mode property is not in cassandra.yaml (looking
> into trunk right now)
>
> I think there's a prehistoric reason why it was removed but I can't
> remember right now.
>
> > Do you all think we can add it there with brief explanation what each
> option does?
>
> We could reinclude it as long as we provide a clear recommendation on when
> to change from the default since this is an advanced setting which should
> be rarely changed. But I still think we should provide a more
> stable/foolproof default (mmap_index_only) since the current default (mmap)
> is known to cause instability in some scenarios.
>
> Also there is a technicality with changing the default, if we change the
> "auto" behavior from mmap to mmap_index_only this may affect users relying
> on the default "mmap" behavior. Not sure the best way to address that, is a
> big NEWS note sufficient? Even though users are expected to read NEWS when
> upgrading we know well not all users read it.
>
> > Shall we also share this thread with @user?
>
> Thanks Ekaterina! If we decide to change the default we can run this
> through the user@ list to see what the user community thinks.
>
> On Wed, Sep 6, 2023 at 4:45 PM Ekaterina Dimitrova 
> wrote:
>
>> Thanks for starting this discussion, Paulo!
>>
>> Shall we also share this thread with @user?
>>
>> On Wed, 6 Sep 2023 at 16:35, C. Scott Andreas 
>> wrote:
>>
>>> Supportive of switching the default to mmap_index_only as well.
>>>
>>> I don’t have numbers handy to share, but my experience has been
>>> significantly lower read latency and I wouldn’t run with auto. I’ve also
>>> not observed substantial heap pressure after switching - it was strictly an
>>> improvement.
>>>
>>> - Scott
>>>
>>> —
>>> Mobile
>>>
>>> On Sep 6, 2023, at 8:50 AM, Paulo Motta 
>>> wrote:
>>>
>>> 
>>>
>>> Hi,
>>>
>>> I've been bitten by OOMs with disk_access_mode:auto/mmap that were fixed
>>> by changing to disk_access_mode:mmap_index_only. In a particular benchmark
>>> I got 5x more read throughput on 3.11.x with disk_access_mode:
>>> mmap_index_only vs disk_access_mode: auto/mmap.
>>>
>>> Changing disk_access_mode to mmap_index_only seems to be a common
>>> recommendation on forums[1][2][3][4] and slack (find by searching
>>> disk_access_mode in the #cassandra channel on https://the-asf.slack.com/
>>> ).
>>>
>>> It's not clear to me when using the default
>>> disk_access_mode:auto/mmap is beneficial, perhaps only when the read set
>>> fits in memory? Mick seems to think on CASSANDRA-15531 [5], that
>>> mmap_index_only has a higher heap cost and should be only used when
>>> warranted. However it's not uncommon to see people being bitten with OOMs
>>> or lower read performance due to the default disk_access_mode, so it makes
>>> me think it's not the best fool-proof default.
>>>
>>> Should we consider changing default "auto" behavior of
>>> "disk_access_mode" to be "mmap_index_only" instead of "mmap" in 5.0 since
>>> it's likely safer and perhaps more performant?
>>>
>>> Thanks,
>>>
>>> Paulo
>>>
>>> [1]
>>> https://stackoverflow.com/questions/72272035/troubleshooting-and-fixing-cassandra-oom-issue
>>> [2] https://phabricator.wikimedia.org/T137419
>>> [3] https://stackoverflow.com/a/55975471
>>> [4]
>>> https://support.datastax.com/s/article/FAQ-Use-of-disk-access-mode-in-DSE-51-and-earlier
>>> [5] https://issues.apache.org/jira/browse/CASSANDRA-15531
>>>
>>>