This is the approach I favour for config files also. We had a much less engaged
discussion on this topic only a few months ago, so glad to see more people
getting involved now.
I would however personally prefer to see the configuration file slowly
deprecated (if perhaps never retired), in favour of virtual tables, so that
operators may easily set configurations for the entire cluster. Ideally it
would be possible to specify configuration per cluster, per DC and per node,
with the most specific configuration applying I would like to see a similar
hierarchy for Keyspace, Table and Per-Query options. Ideally only the barest
minimum number of options would be necessary to supply in a config file, and
only on first launch – seed nodes, for instance.
So whatever design we employ here, we should IMO be aiming for it to be
compatible with a CQL representation also.
From: Bowen Song <bo...@bso.ng.INVALID>
Date: Wednesday, 24 November 2021 at 18:15
To: dev@cassandra.apache.org <dev@cassandra.apache.org>
Subject: Re: [DISCUSS] Nested YAML configs for new features
Since you mentioned ElasticSearch, I'm actually pretty happy with their
config file syntax. It allows the user to completely flatten out the
entire config file. To give people who isn't familiar with ElasticSearch
an idea, here is a config file we use:
cluster.name: foobar
node.remote_cluster_client: false
node.name: "foo.example.com"
node.master: true
node.data: true
node.ingest: true
node.ml: false
xpack.ml.enabled: false
xpack.security.enabled: false
xpack.security.audit.enabled: false
xpack.watcher.enabled: false
action.auto_create_index: "+.,-*"
network.host: _global_
discovery.zen.hosts_provider: file
discovery.zen.minimum_master_nodes: 2
http.publish_host: "foo.example.com"
http.publish_port: 443
http.bind_host: 127.0.0.1
transport.publish_host: "bar.example.com"
transport.bind_host: 0.0.0.0
indices.fielddata.cache.size: 1GB
indices.breaker.total.use_real_memory: false
path.logs: /var/log/elasticsearch
path.data: /var/lib/elasticsearch/data
As you can see we can use the flat (grep-able) syntax for everything.
This is also human readable because we can group options together by
inserting empty lines between them.
The equivalent of the above in a structured syntax will be:
cluster:
name: foobar
node:
remote_cluster_client: false
name: "foo.example.com"
master: true
data: true
ingest: true
ml: false
xpack:
ml:
enabled: false
security:
enabled: false
audit:
enabled: false
watcher:
enabled: false
action:
auto_create_index: "+.,-*"
network:
host: _global_
discovery:
zen:
hosts_provider: file
minimum_master_nodes: 2
http:
publish_host: "foo.example.com"
publish_port: 443
bind_host: 127.0.0.1
transport:
publish_host: "bar.example.com"
bind_host: 0.0.0.0
indices:
fielddata:
cache:
size: 1GB
indices:
breaker:
total:
use_real_memory: false
path:
logs: /var/log/elasticsearch
data: /var/lib/elasticsearch/data
This may be easier to read for some people, but it is a total nightmare
for "grep" - so many keys have identical names, such as "enabled".
Also, for the virtual tables, it would be a lot easier to represent
individual values in a virtual table when the config is flat and keys
are unique. The virtual tables would need to either support the encoding
and decoding of the structured config into a flat structure, or use JSON
encoded string value. The use of JSON would make querying individual
value much harder.
On 22/11/2021 16:16, Joseph Lynch wrote:
> Isn't one of the primary reasons to have a YAML configuration instead
> of a properties file is to allow typed and structured (implies nested)
> configuration? I think it makes a lot of sense to group related
> configuration options (e.g. a feature) into a typed class when we're
> talking about more than one or two related options.
>
> It's pretty standard elsewhere in the JVM ecosystem to encode YAMLs to
> period encoded key->value pairs when required (usually when providing
> a property or override layer), Spring and Elasticsearch yamls both
> come to mind. It seems pretty reasonable to support dot encoding and
> decoding, for example {"a": {"b": 12}} -> '"a.b": 12'.
>
> Regarding quickly telling what configuration a node is running I think
> we should lean on virtual tables for "what is the current
> configuration" now that we have them, as others have said the written
> cassandra.yaml is not necessarily the current configuration ... and
> also grep -C or -A exist for this reason.
>
> -Joey
>
> On Mon, Nov 22, 2021 at 4:14 AM Benjamin Lerer<ble...@apache.org> wrote:
>> I do not have a strong opinion for one or the other but wanted to raise the
>> issue I see with the "Settings" virtual table.
>>
>> Currently the "Settings" virtual table converts nested options into flat
>> options using a "_" separator. For those options it allows a user to query
>> the all set of options through some hack.
>> If we decide to move to more nesting (more than one level), it seems to me
>> that we need to change the way this table is behaving and how we can query
>> its data.
>>
>> We would need to start using "." as a nesting separator to ensure that
>> things are consistent between the configuration and the table and add
>> support for LIKE restrictions for filtering queries to allow operators to
>> be able to select the precise set of settings that the operator is looking
>> for.
>>
>> Doing so is not really complicated in itself but might impact some users.
>>
>> Le ven. 19 nov. 2021 à 22:39, David Capwell<dcapw...@apple.com.invalid> a
>> écrit :
>>
>>>> it is really handy to grep
>>>> cassandra.yaml on some config key and you know the value instantly.
>>> You can still do that
>>>
>>> $ grep -A2 coordinator_read_size conf/cassandra.yaml
>>> # coordinator_read_size:
>>> # warn_threshold_kb: 0
>>> # abort_threshold_kb: 0
>>>
>>> I was also arguing we should support nested and flat, so if your infra
>>> works better with flat then you could use
>>>
>>> track_warnings.coordinator_read_size.warn_threshold_kb: 0
>>> track_warnings.coordinator_read_size.abort_threshold_kb: 0
>>>
>>>> On Nov 19, 2021, at 1:34 PM, David Capwell<dcapw...@apple.com> wrote:
>>>>
>>>>> With the flat structure it turns into properties file - would it be
>>>>> possible to support both formats - nested yaml and flat properties?
>>>>
>>>> For majority of our configs yes, but there are a subset where flat
>>> properties is annoying
>>>> hinted_handoff_disabled_datacenters - set type, so you could do
>>> hinted_handoff_disabled_datacenters=“a,b,c,d” but we would need to deal
>>> with separators as the format doesn’t support
>>>> seed_provider.parameters - this is a map type… so would need to do
>>> something like seed_provider.parameters=“{\”a\”: \a\”}” …. Maybe we special
>>> case maps as dynamic fields? Then seed_provider.parameters.a=a? We have
>>> ParameterizedClass all over the code
>>>> So, as long as we define how to deal with java collections; we could in
>>> theory support properties files (not arguing for that in this thread) as
>>> well as system properties.
>>>>
>>>>> On Nov 19, 2021, at 1:22 PM, Jacek Lewandowski <
>>> lewandowski.ja...@gmail.com> wrote:
>>>>> With the flat structure it turns into properties file - would it be
>>>>> possible to support both formats - nested yaml and flat properties?
>>>>>
>>>>>
>>>>> - - -- --- ----- -------- -------------
>>>>> Jacek Lewandowski
>>>>>
>>>>>
>>>>> On Fri, Nov 19, 2021 at 10:08 PM Caleb Rackliffe <
>>> calebrackli...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> If it's nested, "track_warnings" would still work if you're grepping
>>> around
>>>>>> vim or less.
>>>>>>
>>>>>> I'd have to concede the point about grep output, although there are
>>> tools
>>>>>> likehttps://github.com/kislyuk/yq that could probably be bent to do
>>> what
>>>>>> you want.
>>>>>>
>>>>>> On Fri, Nov 19, 2021 at 1:08 PM Stefan Miklosovic <
>>>>>> stefan.mikloso...@instaclustr.com> wrote:
>>>>>>
>>>>>>> Hi David,
>>>>>>>
>>>>>>> while I do not oppose nested structure, it is really handy to grep
>>>>>>> cassandra.yaml on some config key and you know the value instantly.
>>>>>>> This is not possible when it is nested (easily & fastly) as it is on
>>>>>>> two lines. Or maybe my grepping is just not advanced enough to cover
>>>>>>> this case? If it is flat, I can just grep "track_warnings" and I have
>>>>>>> them all.
>>>>>>>
>>>>>>> Can you elaborate on your last bullet point? Parsing layer ... What do
>>>>>>> you mean specifically?
>>>>>>>
>>>>>>> Thanks
>>>>>>>
>>>>>>> On Fri, 19 Nov 2021 at 19:36, David Capwell<dcapw...@gmail.com>
>>> wrote:
>>>>>>>> This has been brought up in a few tickets, so pushing to the dev
>>> list.
>>>>>>>> CASSANDRA-15234 - Standardise config and JVM parameters
>>>>>>>> CASSANDRA-16896 - hard/soft limits for queries
>>>>>>>> CASSANDRA-17147 - Guardrails prototype
>>>>>>>>
>>>>>>>> In short, do we as a project wish to move "new features" into nested
>>>>>>>> YAML when the feature has "enough" to justify the nesting? I would
>>>>>>>> really like to focus this discussion on new features rather than
>>>>>>>> retroactively grouping (leaving that to CASSANDRA-15234), as there is
>>>>>>>> already a place to talk about that.
>>>>>>>>
>>>>>>>> To get things started, let's start with the track-warning feature
>>>>>>>> (hard/soft limits for queries), currently the configs look as follows
>>>>>>>> (assuming 15234)
>>>>>>>>
>>>>>>>> track_warnings:
>>>>>>>> enabled: true
>>>>>>>> coordinator_read_size:
>>>>>>>> warn_threshold: 10kb
>>>>>>>> abort_threshold: 1mb
>>>>>>>> local_read_size:
>>>>>>>> warn_threshold: 10kb
>>>>>>>> abort_threshold: 1mb
>>>>>>>> row_index_size:
>>>>>>>> warn_threshold: 100mb
>>>>>>>> abort_threshold: 1gb
>>>>>>>>
>>>>>>>> or should this be "flat"
>>>>>>>>
>>>>>>>> track_warnings_enabled: true
>>>>>>>> track_warnings_coordinator_read_size_warn_threshold: 10kb
>>>>>>>> track_warnings_coordinator_read_size_abort_threshold: 1mb
>>>>>>>> track_warnings_local_read_size_warn_threshold: 10kb
>>>>>>>> track_warnings_local_read_size_abort_threshold: 1mb
>>>>>>>> track_warnings_row_index_size_warn_threshold: 100mb
>>>>>>>> track_warnings_row_index_size_abort_threshold: 1gb
>>>>>>>>
>>>>>>>> For me I prefer nested for a few reasons
>>>>>>>> * easier to enforce consistency as the configs can use shared types;
>>>>>>>> in the track warnings patch I had mismatches cross configs (warn vs
>>>>>>>> warns, fail vs abort, etc.) before going nested, now everything
>>> reuses
>>>>>>>> the same types
>>>>>>>> * even though it is longer, things can be more clear how they are
>>>>>> related
>>>>>>>> * parsing layer can add support for mixed or purely flat depending on
>>>>>>>> user preference (example:
>>>>>>>> track_warnings.row_index_size.abort_threshold, using the '.' notation
>>>>>>>> to represent nested structures)
>>>>>>>>
>>>>>>>> Thoughts?
>>>>>>>>
>>>>>>>> ---------------------------------------------------------------------
>>>>>>>> To unsubscribe, e-mail:dev-unsubscr...@cassandra.apache.org
>>>>>>>> For additional commands, e-mail:dev-h...@cassandra.apache.org
>>>>>>>>
>>>>>>> ---------------------------------------------------------------------
>>>>>>> To unsubscribe, e-mail:dev-unsubscr...@cassandra.apache.org
>>>>>>> For additional commands, e-mail:dev-h...@cassandra.apache.org
>>>>>>>
>>>>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail:dev-unsubscr...@cassandra.apache.org
>>> For additional commands, e-mail:dev-h...@cassandra.apache.org
>>>
>>>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:dev-unsubscr...@cassandra.apache.org
> For additional commands, e-mail:dev-h...@cassandra.apache.org
>
