LebronAl edited a comment on issue #3954: URL: https://github.com/apache/iotdb/issues/3954#issuecomment-918760291
> Agree that the case is a bug for IoTDB. Yes > Of course, this is strictly an implementation bug and can be fixed. But even if the bug is fixed, you can see that our raft log's serialization buffer refers to the implementation of the stand-alone WAL, it does not write to the disk and call async every time a log is written, which ensures that the performance will not be limited by the IOPS of the disk. But this may also cause the corner case mentioned above. This is the trade-off between performance and safety. But the key sentence is this. Even if the bug is fixed. Should we also call async every time when we flush log? The current implementation is a performance-oriented compromise for us, but it certainly sacrifices security. Strictly speaking, we currently buffer data through user-space buffers, whether standalone wal or raft logs. However, with [Ratis](https://github.com/apache/ratis/blob/7a852736de1588bc645f86a92a1d6b4d37b21e2b/ratis-server/src/main/java/org/apache/ratis/server/raftlog/segmented/SegmentedRaftLogWorker.java#L510), they seem to be caching the data in the operating system's Page cache, and `async` is determined based on the parameters and load. I think our approach’s performance will be better (because it sacrifices more security than theirs), but I don't know how much of performance difference there is. This is worth exploring. > And let's go back to Raft library topic. Import the Raft library doesn't mean application needs to sync all operation via the library, it depends on requirement. I don't think IoTDB needs to change current read action process flow after import the library. Hope so. IMO, the more additional code we have to write using these Raft libraries, the more optimization possibilities we have for our scenario, just like with etcd. But with Ratis, I'm not sure yet. > Agree the concept but which is unnecessary under IoTDB or time-order data environment. I think the exactly-only-once requirement here as the actions are not idempotent for all system. In fact, even if every command is idempotent, as long as the client adds retry logic, it will violate linearizability without filtering as described above... This is a fairly advanced feature, as far as I know, only the Raft library for Dragonboat supports this logic, and I don't think Ratis has done it yet. So we can forget about this. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
