Ok I understand now. How do you specify what goes where? Since the raft message is just a byte string and can't specify what is what.
In the filestoreclient it sends the bytes of a proto and in the statemachine the code just assumes that the logdata and statemachinedata are in the right places? How is that being done? ________________________________ From: Tsz Wo Sze <[email protected]> Sent: Monday, December 20, 2021 1:07:55 AM To: [email protected] <[email protected]> Subject: Re: Raft log persistance > ... what is the difference between StateMachineLogEntryProto#logData and > StateMachineLogEntryProto#stateMachineEntry#stateMachineData ? StateMachineLogEntryProto#logData is the data that will be written to RaftLog. In contrast, StateMachineLogEntryProto#stateMachineEntry#stateMachineData won't be written to the RaftLog. The state machine is going to store it. > In the counter example the log data is the 'increment request' and in > filestore the stateMachineData is the stateMachineLogEntry is being used? In the counter example, the log data contains all the data in the request and the request is just the "increment request". In filestore, a client may send a request to write a file containing data D to a particular path P. The stateMachineData is the file data D and the log data is the meta data such as file path P. > What should be used where/when? StateMachineLogEntryProto#logData is the logData specified in the RAFT algorithm. StateMachineLogEntryProto#stateMachineEntry#stateMachineData is the data that the state machine wants to store the data itself only but not in the RaftLog in order to avoid storing two copies of data. Please feel free to let me know if you have more questions. Tsz-Wo On Sat, Dec 18, 2021 at 11:42 PM Asad Awadia <[email protected]<mailto:[email protected]>> wrote: I think I am starting to understand. Reading the raft.proto file - what is the difference between StateMachineLogEntryProto#logData and StateMachineLogEntryProto#stateMachineEntry#stateMachineData ? In the counter example the log data is the 'increment request' and in filestore the stateMachineData is the stateMachineLogEntry is being used? What should be used where/when? ________________________________ From: Tsz Wo Sze <[email protected]<mailto:[email protected]>> Sent: Saturday, December 18, 2021 1:36:18 AM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: Raft log persistance The file store example is just a simple example. It does not intend to support all the features. It indeed does not work for the case that the data is deleted -- it will fail to read the file. You are right that, for real applications supporting multiple versions of data, it should retrieve the data from the version corresponding to the given log index. Tsz-Wo On Sat, Dec 18, 2021 at 1:05 PM Asad Awadia <[email protected]<mailto:[email protected]>> wrote: In file store example read method - it looks like it is returning the file at the path without taking the raft index in consideration i.e it would keep returning the same file for many entry indices provided the file hadn't changed? What is going on there? Shouldn't it return the data snapshot at instant entry.getIndex? ________________________________ From: Tsz Wo Sze <[email protected]<mailto:[email protected]>> Sent: Wednesday, December 15, 2021 7:37:20 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: Raft log persistance > If I understand correctly, in the write method we should take the request > and apply it right there in the method into the state machine? So it only > gets persisted into the state machine and not the raft log? The write method should write the stateMachineData (i.e. LogEntryProto.getStateMachineLogEntry().getStateMachineEntry().getStateMachineData()) to its storage. The stateMachineData will be removed from the logEntryProto. The logEntryProto with removed stateMachineData will be written to the raft log. Yes, stateMachineData won't be written to the raft log. > How will syncing up and backups work then? If there is no 'raft log' anymore > since we are overriding the write method to only apply writes to the state > machine. How will the other servers be able to sync and catch up? When reading from the raft log, the server in addition calls the DataApi.read(LogEntryProto) method. The state machine should return back the stateMachineData corresponding to the given logEntryProto as a CompletableFuture<ByteString>. Then, the server will add the stateMachineData back to the logEntryProto and then send the logEntryProto to the other servers. Tsz-Wo On Thu, Dec 16, 2021 at 3:13 AM Asad Awadia <[email protected]<mailto:[email protected]>> wrote: I read through the file store example. If I understand correctly, in the write method we should take the request and apply it right there in the method into the state machine? So it only gets persisted into the state machine and not the raft log? How will syncing up and backups work then? If there is no 'raft log' anymore since we are overriding the write method to only apply writes to the state machine. How will the other servers be able to sync and catch up? ________________________________ From: Tsz Wo Sze <[email protected]<mailto:[email protected]>> Sent: Tuesday, December 14, 2021 8:16:23 PM To: [email protected]<mailto:[email protected]> <[email protected]<mailto:[email protected]>> Subject: Re: Raft log persistance Hi Asad, By default, all the write requests (including the data inside) are recorded in the raft log. This is specified in the standard RAFT algorithm so that any server can replay the log in order to sync up to the current state. If the state machine keeps another persistent copy of the data, you are right that there will be two copies. In this sense, the standard RAFT algorithm is not suitable for data intensive applications. In Ratis, we want to support data intensive applications such as Apache Ozone. State machine implementations may choose to manage the data itself. In that case, the data won't be recorded in the raft log. In order to use this feature, state machine implementations must override the read and write methods defined in StateMachine.DataApi. The FileStore example indeed has overridden these two methods. Hope it helps. Tsz-Wo On Wed, Dec 15, 2021 at 3:40 AM Asad Awadia <[email protected]<mailto:[email protected]>> wrote: Hello, Is ratis keeping a copy of all the writes made? On top of what is persisted in my own DB? So ~2x storage used I read that the file system example avoids that but I don't see how that is being done in the code? Regards, Asad
