[
https://issues.apache.org/jira/browse/CASSANDRA-18084?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17642845#comment-17642845
]
Stefan Miklosovic edited comment on CASSANDRA-18084 at 12/3/22 2:48 PM:
------------------------------------------------------------------------
Just to comment on implementation we may do here (not saying right now if we do
that or not eventually), in order to have this change minimally disruptive, we
do not need to expose this to any CQL tables. One problem less. It does not
need to be query-able via CQL. It might be exposed via IFailureDetectorMBean
which has bunch of getAllEndpointStates methods. This is called in nodetool
gossipinfo too. So it would be nowhere to find but there.
To load it, as we do not want to put it into rackdc.properties, it might be set
in a system property. Ideally the loader of these properties would be interface
so it could be loaded from wherever but I don't consider it too important in
the light of more mature implementation coming (the cep).
To hold the state we would need to be smart about that, currently Gossip reacts
by persisting them to system.local/peers_v2 primarily, we would either save it
directly in failure detector or in something custom fd would call.
I think that FailureDetector is somewhat unnatural place to put that in, not
super appropriate name of the bean to retrieve that information from but that
is just the consequence of having all gossip-related information retrievable
from there.
New application state would be still added though, that one would be eventually
deprecated. If CEP is in place which would implement this more robustly I think
that we might deprecate this quite easily. I do not think we will ever remove
it from ApplicationState enum, probably not. But what I got from Sam's response
is that they plan to move some things out from Gossip so in that case how would
ApplicationState enum look like? The properties which are currently there would
be eventually removed anyway, no? Or they would stay there?
Anyway, I still feel like hiding that under IFailureDetector's
getAllEndpointStates method is still not something Sam will be particularly fan
of. I am curious what he thinks. I am trying to do in such a way that it will
be easy to deprecate so I am trying to avoid some completely new mbeans and
putting stuff to dedicated package / classes, you feel me.
Or we can go full-on here, to do some "MetadataService" which would be
receptive to Gossip's tags but then I do not know how to deprecate and/or
remove it entirely. I want to avoid situation we need to support this forever.
Maybe MetadataService would be flexible enough to be tweaked in such a way that
Sam's CEP would reuse that to push more stuff through that so we would just
build on top of it rather than trying to completely get rid of it when new
stuff comes.
was (Author: smiklosovic):
Just to comment on implementation we may do here (not saying right now if we do
that or not eventually), in order to have this change minimally disruptive, we
do not need to expose this to any CQL tables. One problem less. It does not
need to be query-able via CQL. It might be exposed via IFailureDetectorMBean
which has bunch of getAllEndpointStates methods. This is called in nodetool
gossipinfo too. So it would be nowhere to find but there.
To load it, as we do not want to put it into rackdc.properties, it might be set
in a system property. Ideally the loader of these properties would be interface
so it could be loaded from wherever but I don't consider it too important in
the light of more mature implementation coming (the cep).
To hold the state we would need to be smart about that, currently Gossip reacts
by persisting them to system.local/peers_v2 primarily, we would either save it
directly in failure detector or in something custom fd would call.
I think that FailureDetector is somewhat unnatural place to put that in, not
super appropriate name of the bean to retrieve that information from but that
is just the consequence of having all gossip-related information retrievable
from there.
New application state would be still added though, that one would be eventually
deprecated. If CEP is in place which would implement this more robustly I think
that we might deprecate this quite easily. I do not think we will ever remove
it from ApplicationState enum, probably not. But what I got from Sam's response
is that they plan to move some things out from Gossip so in that case how would
ApplicationState enum look like? The properties which are currently there would
be eventually removed anyway, no? Or they would stay there?
Anyway, I still feel like hiding that under IFailureDetector's
getAllEndpointStates method is still not something Sam will be particularly fan
of. I am curious what he thinks. I am trying to do in such a way that it will
be easy to deprecate so I am trying to avoid some completely new mbeans and
putting stuff to dedicated package / classes, you feel me.
> Introduce tags to snitch for better decision making for replica placement in
> topology strategies
> ------------------------------------------------------------------------------------------------
>
> Key: CASSANDRA-18084
> URL: https://issues.apache.org/jira/browse/CASSANDRA-18084
> Project: Cassandra
> Issue Type: New Feature
> Components: Cluster/Gossip, Cluster/Schema, Legacy/Distributed
> Metadata
> Reporter: Stefan Miklosovic
> Assignee: Stefan Miklosovic
> Priority: Normal
>
> We would like to have extra meta-information in cassandra-rackdc.properties
> which would further differentiate nodes in dc / racks.
> The main motivation behind this is that we have special requirements around
> node's characteristics based on which we want to make further decisions when
> it comes to replica placement in topology strategies. (New topology strategy
> would be mostly derived from NTS / would be extended)
> The most reasonable way to do that is to introduce a new property into
> cassandra-rackdc.properties called "tags"
> {code:java}
> # Arbitrary tag to assign to this node, they serve as additional
> identificator of a node based on which operators might act.
> # Value of this property is meant to be a comma-separated list of strings.
> #tags=tag1,tag2
> {code}
> We also want to introduce new application state called TAGS. On startup of a
> node, this node would advertise its tags to cluster and vice versa, all nodes
> would tell to that respective node what tags they have so everybody would see
> the same state of tags across the cluster based on which topology strategies
> would do same decisions.
> These tags are not meant to be changed during whole runtime of a node,
> similarly as datacenter and rack is not.
> For performance reasons, we might limit the maximum size of tags (sum of
> their lenght), to be, for example, 64 characters and anything bigger would be
> either shortened or the start would fail.
> Once we have tags for all nodes, we can have access to them, cluster-wide,
> from TokenMetadata which is used quite heavily in topology strategies and it
> exposes other relevant topology information (dc's, racks ...). We would just
> add a new way to look at nodes.
> Tags would be a set.
> This would be persisted to the system.local to see what tags a local node has
> and it would be persisted to system.peers_v2 to see what tags all other nodes
> have. Column would be called "tags".
> {code:java}
> admin@cqlsh> select * from system.local ;
> @ Row 1
> key | local
> bootstrapped | COMPLETED
> broadcast_address | 172.19.0.5
> broadcast_port | 7000
> cluster_name | Test Cluster
> cql_version | 3.4.6
> data_center | dc1
> gossip_generation | 1669739177
> host_id | 54f8c6ea-a6ba-40c5-8fa5-484b2b4184c9
> listen_address | 172.19.0.5
> listen_port | 7000
> native_protocol_version | 5
> partitioner | org.apache.cassandra.dht.Murmur3Partitioner
> rack | rack1
> release_version | 4.2-SNAPSHOT
> rpc_address | 172.19.0.5
> rpc_port | 9042
> schema_version | ef865449-2491-33b8-95b0-47c09cb14ea9
> tags | {'tag1', 'tag2'}
> tokens | {'6504358681601109713'}
> {code}
> for system.peers_v2:
> {code:java}
> admin@cqlsh> select peer,tags from system.peers_v2 ;
> @ Row 1
> ------+-----------------
> peer | 172.19.0.15
> tags | {'tag2', 'tag3'}
> @ Row 2
> ------+-----------------
> peer | 172.19.0.11
> tags | null
> {code}
> the POC implementation doing exactly that is here:
> We do not want to provide our custom topology strategies which are using this
> feature as that will be the most probably a proprietary solution. This might
> indeed change in the future. For now, we just want to implement hooks we can
> base our in-house implementation on. All other people can benefit from this
> as well if they choose so as this feature enables them to do that.
> Adding tags is not only about custom topology strategies. Operators could tag
> their nodes if they wish to make further distinctions on topology level for
> their operational needs.
> [https://github.com/instaclustr/cassandra/commit/eddd4681d8678515454dabb926d5f56b0c225eea]
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscr...@cassandra.apache.org
For additional commands, e-mail: commits-h...@cassandra.apache.org
