This is an automated email from the ASF dual-hosted git repository.
hanahmily pushed a commit to branch main
in repository https://gitbox.apache.org/repos/asf/skywalking-banyandb.git
The following commit(s) were added to refs/heads/main by this push:
new 147972af Add file system vector-IO operations interface. (#312)
147972af is described below
commit 147972affe09c95df513ba9e721b6458fa33f735
Author: HHoflittlefish777 <[email protected]>
AuthorDate: Wed Aug 9 20:51:11 2023 +0800
Add file system vector-IO operations interface. (#312)
* Add Vector-IO operations interface.
---------
Co-authored-by: 吴晟 Wu Sheng <[email protected]>
Co-authored-by: Gao Hongtao <[email protected]>
---
docs/concept/persistence-storage.md | 21 +++++++++++++++++++--
pkg/fs/file_system.go | 4 ++++
2 files changed, 23 insertions(+), 2 deletions(-)
diff --git a/docs/concept/persistence-storage.md
b/docs/concept/persistence-storage.md
index fdb58716..1b389e91 100644
--- a/docs/concept/persistence-storage.md
+++ b/docs/concept/persistence-storage.md
@@ -12,6 +12,7 @@ For index and data, the architecture of the file system is
divided into three la
- The first layer is the API interface, which developers only need to care
about how to operate the remote file system.
- The second layer is the storage system adapter, which is used to mask the
differences between different storage systems.
- The last layer is the actual storage system. With the use of remote storage
architecture, the local system can still play its role and can borrow the local
system to speed up reading and writing.
+
# IO Mode
@@ -112,7 +113,7 @@ return: File pointer, you can use it for various operations.
### Write
BanyanDB provides two methods for writing files.
-Append mode, which adds new data to the end of a file. This mode is typically
used for WAL.
+Append mode, which adds new data to the end of a file. This mode is typically
used for WAL. And BanyanDB supports vector Append mode, which supports
appending consecutive buffers to the end of the file.
Flush mode, which flushes all data to one file. It will return an error when
writing a directory, the file does not exist or there is not enough space, and
the incomplete file will be discarded. The flush operation is atomic, which
means the file won't be created if an error happens during the flush process.
The following is the pseudocode that calls the API in the go style.
@@ -124,6 +125,14 @@ buffer: The data append to the file.
`File.AppendWriteFile(buffer []byte) (error)`
+For vector append mode:
+
+param:
+
+iov: The data in consecutive buffers.
+
+`File.AppendWritevFile(iov *[][]byte) (error)`
+
For flush mode:
param:
@@ -145,7 +154,7 @@ The following is the pseudocode that calls the API in the
go style.
### Read
For reading operation, two read methods are provided:
-Reading a specified location of data, which relies on a specified offset and a
buffer.
+Reading a specified location of data, which relies on a specified offset and a
buffer. And BanyanDB supports reading contiguous regions of a file and
dispersing them into discontinuous buffers.
Read the entire file, BanyanDB provides stream reading, which can use when the
file is too large, the size gets each time can be set when using stream reading.
If entering incorrect parameters such as incorrect offset or non-existent
file, it will return an error.
The following is the pseudocode that calls the API in the go style.
@@ -160,6 +169,14 @@ buffer: The read length is the same as the buffer length.
`File.ReadFile(offset int, buffer []byte) (error)`
+For vector reading:
+
+param:
+
+iov: Discontinuous buffers in memory.
+
+`File.ReadvFile(iov *[][]byte) (error)`
+
For stream reading:
param:
diff --git a/pkg/fs/file_system.go b/pkg/fs/file_system.go
index eb61eecc..8003cf8a 100644
--- a/pkg/fs/file_system.go
+++ b/pkg/fs/file_system.go
@@ -44,10 +44,14 @@ type Dir interface {
type File interface {
// Append mode, which adds new data to the end of a file.
AppendWriteFile(buffer []byte) error
+ // Vector Append mode, which supports appending consecutive buffers to
the end of the file.
+ AppendWritevFile(iov *[][]byte) error
// Delete the file.
DeleteFile() error
// Reading a specified location of file.
ReadFile(offset int, buffer []byte) error
+ // Reading contiguous regions of a file and dispersing them into
discontinuous buffers.
+ ReadvFile(iov *[][]byte) error
// Read the entire file using streaming read
StreamReadFile(offset int, buffer []byte) (*Iter, error)
// Rename the file.
