EvanLjp commented on a change in pull request #10: URL: https://github.com/apache/skywalking-satellite/pull/10#discussion_r545725086
########## File path: docs/plugins/queue/mmap/README.md ########## @@ -0,0 +1,66 @@ +# Design +The mmap-queue is a big, fast and persistent queue based on the memory mapped files. One mmap-queue has a directory to store the whole data. The Queue directory is made up with many segments and 1 meta file. + +- Segment: Segment is the real data store center, that provides large-space storage and does not reduce read and write performance as much as possible by using mmap. And we will avoid deleting files by reusing them. +- Meta: The purpose of meta is to find the data that the consumer needs. + +## Meta +Metadata only needs 80B to store the Metadata for the pipe. But for memory alignment, it takes at least one memory page size, which is generally 4K. +``` +[ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ][ 8Bit ] +[metaVersion][ ID ][ offset][ ID ][ offset][ ID ][ offset][ ID ][ offset][capacity] +[metaVersion][writing offset][watermark offset][committed offset][reading offset][capacity] + +``` +### Transforming + + + +## Configuration +[Configuration Params](../../../configuration/queue.md) + +## Segment +Segments are a series of files of the same size. Each input data would cost `8Bit+Len(data)Bit` to store the raw bytes. The first 8Bit is equal to `Len(data)` for finding the ending position. +### Swapper +For the performance and resources thinking, we define a page replacement policy. + +- Keep the reading and writing segments on the memory. +- When the mmapcount is greater or equals to the max_in_mem_segments, we first scan the read scope and then scan the written scope to swap the segments to promise the reading or writing segments are always in memory. + - Read scope: [reading_offset - max_in_mem_segments,reading_offset - 1] + - Written scope: [writing_offset - max_in_mem_segments,writing_offset - 1] + - Each displacement operation guarantees at least `max_in_mem_segments/2-1` capacity available. Subtract operation to subtract the amount of memory that must always exist. + +## BenchmarkTest Review comment: Yes, you can see in the benchmark test runs over 30000 times, so must trigger the memory swap. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected]
