Ivan Bessonov created IGNITE-26272:
--------------------------------------
Summary: Implement mem-table for incomplete segment files
Key: IGNITE-26272
URL: https://issues.apache.org/jira/browse/IGNITE-26272
Project: Ignite
Issue Type: Improvement
Reporter: Ivan Bessonov
Here we need to have an in-memory structure for offset table of incomplete
segment files. In essence it should look like this:
{code:java}
MemTable = Map<
ReplicationGroupId,
IndexSegmentInfo
>
IndexSegmentInfo = {
flags
startIndex
offsets[]
} {code}
In code it might look something like this:
{code:java}
class IndexSegmentInfo {
volatile int flags;
final int startOffset;
volatile int[] offsets;
volatile int nextOffsetIndex;
}{code}
This structure has to be partitioned into stripes, because it's going to be
used in a disruptor pool and we want as little contention as possible.
Concurrent reads from it should still be possible, that's why there are all
these volatiles.
Append to log storage should be followed with an append to this mem-table.
During the segment switch there must be a way to wait for all stripes in other
thread, in order to collect all mem-table for further asynchronous processing.
It can be expressed with a current pseudo-code:
{code:java}
void append(logEntry) {
while (true) {
segment = currentSegment()
if (!segment.semaphore.acquire())
continue; // Retry, segment had switched in background.
try {
offset = segment.append(logEntry)
if (offset == SEGMENT_SWITCHED)
continue; // Retry.
segment.memTable.append(this, logEntry, offset)
} finally {
segment.semaphore.release()
}
}
}{code}
Of course, "{{{}memTable{}}}" doesn't have to be a field of "{{{}segment{}}}",
this is pseudo-code.
We can use a counter instead of a semaphore, or any other type of primitive
that won't cost much to acquire and release.
Doing the "{{{}segment.semaphore.acquireAll(){}}}" after segment switch
guarantees that all "{{{}segment.memTable.append(...){}}}" have finished their
execution, giving us a fast solution.
Of course, we should have concurrency tests for these all algorithms.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
