Liya Fan created FLINK-12886:
--------------------------------
Summary: Support container memory segment
Key: FLINK-12886
URL: https://issues.apache.org/jira/browse/FLINK-12886
Project: Flink
Issue Type: New Feature
Components: Table SQL / Runtime
Reporter: Liya Fan
Assignee: Liya Fan
Attachments: image-2019-06-18-17-59-42-136.png
We observe that in many scenarios, the operations/algorithms are based on an
array of MemorySegment. These memory segments form a large, combined, and
continuous memory space.
For example, suppose we have an array of n memory segments. Memory addresses
from 0 to segment_size - 1 are served by the first memory segment; memory
addresses from segment_size to 2 * segment_size - 1 are served by the second
memory segment, and so on.
Specific algorithms decide the actual MemorySegment to serve the operation
requests. For some rare cases, two or more memory segments serve the requests.
There are many operations based on such a paradigm, for example,
{{BinaryString#matchAt}}, {{SegmentsUtil#copyToBytes}},
{{LongHashPartition#MatchIterator#get}}, etc.
The problem is that, for memory segment array based operations, large amounts
of code is devoted to
1. Computing the memory segment index & offset within the memory segment.
2. Processing boundary cases. For example, to write an integer, there are only
2 bytes left in the first memory segment, and the remaining 2 bytes must be
written to the next memory segment.
3. Differentiate processing for short/long data. For example, when copying
memory data to a byte array. Different methods are implemented for cases when
1) the data fits in a single segment; 2) the data spans multiple segments.
Therefore, there are much duplicated code to achieve above purposes. What is
worse, this paradigm significantly increases the amount of code, making the
code more difficult to read and maintain. Furthermore, it easily gives rise to
bugs which difficult to find and debug.
To address these problems, we propose a new type of memory segment:
{{ContainerMemorySegment}}. It is based on an array of underlying memory
segments with the same size. It extends from the {{MemorySegment}} base class,
so it provides all the functionalities provided by {{MemorySegment}}. In
addition, it hides all the details for dealing with specific memory segments,
and acts as if it were a big continuous memory region.
A prototype implementation is given below:
!image-2019-06-18-17-59-42-136.png|thumbnail!
With this new type of memory segment, many operations/algorithms can be greatly
simplified, without affecting performance. This is because,
1. Many checks, boundary processing are already there. We just move them to the
new class.
2. We optimize the implementation of the new class, so the special
optimizations (e.g. optimizations for short data) are still preserved.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
