Hi @leo65535, First of all, thank you for your attention, thanks!
I just replied in this issue, https://github.com/apache/incubator-inlong/issues/1599 The Flume component used by DataProxy does not quite meet our needs, and multiple rounds of technical discussions have been conducted internally. But it would be prudent to introduce a heavier component to the access side: the access side environment is not good (such as the machine situation, including the number of cores, memory, CPU, etc., as well as the distributed maintenance of the machine, etc.), the module we focus on stability, performance, and flexible management and control to implement it. We haven’t found a suitable component for this module, and if not found last, we will consider developing it ourselves and if you are interested in this module, we can discuss it together。 Thanks again! leo65535 <[email protected]> 于2021年10月9日周六 上午11:45写道： > hi dev, > > > > > Apache inlong contains four module, includes Ingest, Converge, Cache, > Consume. After looking into the project, we > > can find that there is a similar structure in several modules, we can call > it "source - channel - sink"(SCS in short). > > - Ingest module, there is a customized "source - channel - sink" > structure. > > - DataProxy module, we can see this module depends on flume project, which > is a typical SCS structure. > > - Consume module, we can also think flink is also a SCS structure. > > > > > Several disadvantages: > > In Ingest module, we may need to implement many sources ourselves, > inculdes streaming and batch ingest, like cdc, file, socket... > > In DataProxy module, as we known, the development of flume project is > getting slower. > > > > > We can see that it's hard for us to maintain them, we need to learn the > flume api and implements many connectors. The apache flink already contains > rich connectors, and support sql grammar, batch and streaming source > connectors, also > > contains many other excellent features, we can benefit from it. So, here > we can merge some functions of the Ingest, Converge module and introduce > flink to dataproxy module. > > > > > Here the new architecture diagram compared to origin. > > - Origin > > > https://user-images.githubusercontent.com/91316485/136642894-d7a7fd5c-45a7-46b3-ba72-00066839a176.png > > - New > > > https://user-images.githubusercontent.com/91316485/136642903-89730b52-6b46-4a57-8791-890c72d2327c.png > > > > > Looking forward to your ideas, thanks. > > > > > Best, > > Leo65535
