[
https://issues.apache.org/jira/browse/CASSANDRA-11559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16414721#comment-16414721
]
Alex Lourie commented on CASSANDRA-11559:
-----------------------------------------
I've done some work on this ticket and it's available at
[https://github.com/apache/cassandra/compare/trunk...alourie:CASSANDRA-11559#files_bucket].
Few pointers:
* I've refactored InetAddressAndPort class into VirtualEndpoint class across
all the codebase (this will be responsible for the majority of the code
changes).
* I've added a UUID field to hold the hostID value of the endpoint and added
additional methods for working with it.
* I've reworked the TokenMetadata to hold structures other than maps for
UUID-host references and they would no longer be needed, i.e. keeping just a
set of endpoints is enough to hold both address data and the hostID data and to
also look up hosts by IDs or the vice versa.
* I've reworked the SystemKeyspace to also acknowledge the hostIDs where
significant (in local data/peer data storing/fetching), and also only create
new local id if requested (in most cases only when the node is created for the
first time, but also useful for tests that require initiating multiple "nodes"
on the same machine)
I've added a field in DatabaseDescriptor to mark that SystemKeyspace is ready
to be read. This is required for many unit tests that set up clusters "on the
fly" and for further endpoint information discovery during the test run.
* I've updated required unit tests to properly utilise the new object and
initialise and clean others as required.
* I've updated the code in some other locations to incorporate this change,
which does make it simpler on many occasions.
The current state is everything _seems_ to be working and the unit tests pass
([https://circleci.com/gh/alourie/cassandra/97])
The complication that comes out of this work is with building unit tests - the
host ID would now be kept in multiple structures:
* a VirtualEndpoint object when instantiated.
* SystemKeyspace.localHost (queries the DB)
* SystemKeyspace.peersInfo (queries the DB)
* TokenMetadata lists (such as allEndpoints, tokenMap, etc)
* Gossip.instance.endpointState maps (the specific endpoint is added including
the uuid)
* FBUtilities also keeps local reference once fetched
As a result, when creating tests, one needs to update or clear the
hostID-related information in all relevant places, otherwise, tests would fail
with really confusing messages (in most cases because in some thread an
endpoint comparison will happen and UUIDs won't match), such as "no seeds
found", "host cannot be contacted" or various kinds of timeouts and NPEs.
Additionally, when SystemKeyspace is ready to be read within a test flow, a
DatabaseDescriptor.canReadSystemKeyspace field will need to be set to true so
that the UUID would be fetched from SystemKeyspace.
Additionally, at the moment we are keeping EndpointState separately from this
object (in Gossip). Considering that now this VirtualEndpoint can include
basically any information about the endpoint, it may as well incorporate its
own state, and then all handling of the network/state information about an
endpoint will be in one place. Supposedly this should simplify things further
and allow clearing a lot of code.
[~aweisberg] - you have done the previous move away from InetAddress
representation to InetAddressAndPort, which this current patch changes
considerably. I'd love your feedback on this.
Any and all feedback is very welcome.
> Enhance node representation
> ---------------------------
>
> Key: CASSANDRA-11559
> URL: https://issues.apache.org/jira/browse/CASSANDRA-11559
> Project: Cassandra
> Issue Type: Sub-task
> Components: Distributed Metadata
> Reporter: Paulo Motta
> Assignee: Alex Lourie
> Priority: Minor
>
> We currently represent nodes as {{InetAddress}} objects on {{TokenMetadata}},
> what causes difficulties when replacing a node with the same address (see
> CASSANDRA-8523 and CASSANDRA-9244).
> Since CASSANDRA-4120 we index hosts by {{UUID}} in gossip, so I think it's
> time to move that representation to {{TokenMetadata}}.
> I propose representing nodes as {{InetAddress, UUID}} pairs on
> {{TokenMetadata}}, encapsulated in a {{VirtualNode}} interface, so it will
> backward compatible with the current representation, while still allowing us
> to enhance it in the future with additional metadata (and improved vnode
> handling) if needed.
> This change will probably affect interfaces of internal classes like
> {{TokenMetadata}} and {{AbstractReplicationStrategy}}, so I'd like to hear
> from integrators and other developers if it's possible to change these
> without major hassle or if we need to wait until 4.0.
> Besides updating {{TokenMetadata}} and {{AbstractReplicationStrategy}} (and
> subclasses), we will also need to replace all {{InetAddress}} uses with
> {{VirtualNode.getEndpoint()}} calls on {{StorageService}} and related classes
> and tests. We would probably already be able to replace some
> {{TokenMetadata.getHostId(InetAddress endpoint)}} calls with
> {{VirtualNode.getHostId()}}.
> While we will still be dealing with {{InetAddress}} on {{StorageService}} in
> this initial stage, in the future I think we should pass {{VirtualNode}}
> instances around and only translate from {{VirtualNode}} to {{InetAddress}}
> in the network layer.
> Public interfaces like {{IEndpointSnitch}} will not be affected by this.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
