Hi guys,
Apache Arrow provides a cross-language, standardized, columnar, memory format for data. So it is highly desirable to import Arrow to Flink, and make use of its memory layout and memory management facilities. More background on this can be found in https://issues.apache.org/jira/browse/FLINK-10929 The problem is that, incorporating Arrow to Flink is a big move, and may affect many modules of Flink. In our efforts to vectorize Blink batch jobs, we have tried to incorporate Arrow to Flink. Those were incremental changes, which can be easily turned off with a single flag, and the changes are transparent to other parts of the code base. This is the draft of our design document: https://docs.google.com/document/d/1LstaiGzlzTdGUmyG_-9GSWleKpLRid8SJWXQCTqcQB4/edit?usp=sharing For the first step, I suggest providing a flag which is disabled by default, and let the MemoryManager depend on the Arrow Buffer Allocator. With this change, all the MemorySegment will be based on Arrow buffers, but this is transparent to other components, and should never break them. After this step, we can apply the changes described in the design documents, incrementally. This is our initial thoughts. Would you please give your valuable comments? Best, Liya Fan
