CalvinKirs opened a new issue, #2557:
URL: https://github.com/apache/incubator-seatunnel/issues/2557
WAL (Write Ahead Logging) is generated due to the requirement of master node
data storage persistence. We need to use the file system to complete the node
data persistence.
The node data is a `KV` key-value pair, and the stored data can only be
queried in full. The data is usually in kb-mb size. Direct storage of KV based
on the file system itself is a heavyweight operation, and concurrent writing is
not supported for some file systems. , but in order to ensure efficiency, we
will use batch writing, and how to ensure data accuracy, we use the WAL method.
##WAL's core idea
Usually called write-ahead log, it is a common method to prevent memory
corruption and ensure that data is not lost.
## Outline Design
### WAL structure
We will define a WalEntry as follows
````
class WalEntry{
byte[] key
byte[] value
Type EntryType
}
````
When there is data to write, we will encapsulate the data (single) into a
WalEntry, and then encode it. In addition to the above fields, the encoded data
will also add a CRC value (CRC32 cyclic redundancy check technology, Use 8
bytes to save, so that it can be validated when reading),
### Read and write design
For higher performance, we use continuous append writes to improve
performance, and use a double buffer mechanism, that is, read and write
separation, each with a `buffer`.
````
calss ReadBuffer
class WriterBuffer
````
The written data is first recorded in WAL, and then written to the memory
memtable (the memtable will eventually be flushed into the storage system in
batches)
The size of the WAL file is related to the capacity of the memtable bound to
it. The memtable will have a threshold, and it will be closed when it is full,
and the WAL will not accept new writes at this time, so the capacity of the WAL
file usually does not expand infinitely . After the data in the memtable is
flushed to disk by a background thread, and no other errors occur, the WAL can
be safely deleted.
When SeaTunnel starts, we will fully load the data in the WAL and restore
the full disk.
All write operations are performed in the memtable. When the memtable space
is insufficient, a new memtable will be created to continue to receive write
operations. The original memory will be marked as read-only mode, waiting to be
flushed.
### WAL brush strategy
When data is written to the WAL file, we need to manually call flush to
complete it.
There are usually three strategies in the industry,
Since we can't stand data loss, we only use one strategy - instant flushing,
but also configurable time period flushing.
### How to utilize an existing filesystem
We will split the previous FileStorage, and third-party file system
instances (such as Hdfs) will be split for reuse.
````
public HdfsStorage(Map<String, String> configuration) throws
CheckpointStorageException {
this.initStorage(configuration);
}
````
init storage will be independent for other business calls,
And checkpoint will be used as a storage business type of storage.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
