[
https://issues.apache.org/jira/browse/RATIS-279?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16622836#comment-16622836
]
Sergey Soldatov commented on RATIS-279:
---------------------------------------
bq. These all we can do in LogStateMachine with the configurable directory to
store all the logs and the meta data information of the log files can be get
from the LogStateMachine right?
Actually not. LogStateMachine is an implementation of a single Log (in our case
a single state machine). We may have a bunch of them and they may not know
about each other. And obviously, we should have a some kind of admin protocol
to perform basic operations:
1. get list of all logs regardless of the number of state machines we have and
where they are located (peers)
2. create a new Log with or without specifying the explicit list of peers (this
service might to decide on its own where it will be located. We also may keep
in mind that in the future we may want to add some policies like 'rack
awareness' and develop this code accordingly)
3. get the particular log info (list of peers, UUID)
4. perform close/delete operations on the Log.
5. Some maintanance operations like addNewPeer, changeReplicationFactor could
be useful as well.
More complex operations like get list of logs by pattern, disable log creation
by pattern would be required as well.
How implement that? Well, I was thinking that it should work something like
that:
1. We have a metadataStateMachine (lets call it the master). It keeps track of
available peers (live and dead) and maps between LogName -> GroupID<peers>.
This master state machine lives on the hardcoded nodes (i.e. specified by the
configuration). It persist its state (so we kill all nodes, start them again
and all information is still there)
2. Workers have a RatisServer running with LogStateMachine are able to
register themselves on the master (call with register message at the startup,
so the master would add it to the list of available peers)
3. Edge node may request createLog. That means it create a client for the
master and uses a createLog message. Master checks available peers and using
its own client call addGroup asks them to create a new RatisGroup. Keeps the
mapping between log name and RatisGroup in its own state and return this
information to the edge node. Edge node creates LogServiceClient and further
works with the it to perform the read/write operations.
4. For getLog master would return RatisGroup information for the specific name
operations, like change the replication state or add new peer, work in the same
way - client sends the message to the master, it gets group information from
the internal state, creates a client for it, performs the operation and sends
ack message back to the edge node.
Some things to keep in mind: the master state machine is unique. I.e. it's
running in separate RaftServer instances on a different port ( because worker
and master may be located on the same machine). We need to be sure that we have
all kind of notification (died follower, died the leader, lost the majority)
for the LogStateMachine, so it may act correspondingly.
> Create administrative API for a log stream
> ------------------------------------------
>
> Key: RATIS-279
> URL: https://issues.apache.org/jira/browse/RATIS-279
> Project: Ratis
> Issue Type: Sub-task
> Components: LogService
> Reporter: Josh Elser
> Assignee: Rajeshbabu Chintaguntla
> Priority: Major
> Attachments: RATIS-279_WIP.patch, RATIS-279_v2.patch,
> RATIS-279_v3.patch
>
>
> We need to do basic things like:
> * List all log streams
> * Delete a log stream
> * Truncate a log stream
> This may overlap with functionality that actually should live in HBase.
> Making that distinction is part of the tasks of this issue.
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
