dockerzhang commented on code in PR #404: URL: https://github.com/apache/incubator-inlong-website/pull/404#discussion_r896694361
########## docs/design_and_concept/how_to_extend_extract_or_load_node_en.md: ########## @@ -0,0 +1,216 @@ +--- +title: Sort Plugin +sidebar_position: 3 +--- + +# Overview + +InLong-Sort is known as a real-time ETL system. Currently, supported extract or load includes elasticsearch, HBase, hive, iceberg, JDBC, Kafka, mongodb, mysql, orcale, Postgres, pulsar, etc。InLong-Sort is an ETL solution based on Flink SQL，The powerful expressive power of Flink SQL brings high scalability and flexibility. Basically, the semantics supported by Flink SQL are supported by InLong-Sort。In some scenarios, when the built-in functions of Flink SQL do not meet the requirements, they can also be extended through various UDFs in InLong-Sort. At the same time, it will be easier for those who have used SQL, especially Flink SQL, to get started. + +This article describes how to extend a new source (abstracted as extract node in inlong) or a new sink (abstracted as load node in inlong) in InLong-Sort. After understanding the InLong-Sort architecture, you can understand how the source corresponds to the extract node, and how the sink corresponds to the load node. The architecture of inlong sort can be represented by UML object relation diagram as: + + + +The concepts of each component are: Review Comment: it's better to add a table to show the following item information. ########## docs/design_and_concept/how_to_extend_extract_or_load_node_en.md: ########## @@ -0,0 +1,216 @@ +--- +title: Sort Plugin +sidebar_position: 3 +--- + +# Overview + +InLong-Sort is known as a real-time ETL system. Currently, supported extract or load includes elasticsearch, HBase, hive, iceberg, JDBC, Kafka, mongodb, mysql, orcale, Postgres, pulsar, etc。InLong-Sort is an ETL solution based on Flink SQL，The powerful expressive power of Flink SQL brings high scalability and flexibility. Basically, the semantics supported by Flink SQL are supported by InLong-Sort。In some scenarios, when the built-in functions of Flink SQL do not meet the requirements, they can also be extended through various UDFs in InLong-Sort. At the same time, it will be easier for those who have used SQL, especially Flink SQL, to get started. + +This article describes how to extend a new source (abstracted as extract node in inlong) or a new sink (abstracted as load node in inlong) in InLong-Sort. After understanding the InLong-Sort architecture, you can understand how the source corresponds to the extract node, and how the sink corresponds to the load node. The architecture of inlong sort can be represented by UML object relation diagram as: + + + +The concepts of each component are: + +**Group**: data flow group, including multiple data flows, one group represents one data access + +**Stream**: data flow, a data flow has a specific flow direction + +**GroupInfo**: encapsulation of data flow in sort. a groupinfo can contain multiple dataflowinfo + +**StreamInfo**: abstract of data flow in sort, including various sources, transformations, destinations, etc. of the data flow + +**Node**: abstraction of data source, data transformation and data destination in data synchronization + +**ExtractNode**: source-side abstraction for data synchronization + +**TransformNode**: transformation process abstraction of data synchronization + +**LoadNode**: destination abstraction for data synchronization + +**NodeRelationShip**: abstraction of each node relationship in data synchronization + +**FieldRelationShip**: abstraction of the relationship between upstream and downstream node fields in data synchronization + +**FieldInfo**: node field + +**MetaFieldInfo**: node meta fields + +**Function**: abstraction of transformation function + +**FunctionParam**: input parameter abstraction of function + +**ConstantParam**: constant parameters + +To extend the extract node or load node, you need to do the following: 1. Inherit the node class (such as MyExtractNode) and build specific extract or load usage logic; 2. In a specific node class (such as MyExtractNode), specify the corresponding Flink connector; 3. Use specific node classes in specific ETL implementation logic (such as MyExtractNode) Review Comment: To extend the extract node or load node, you need to do the following: - Inherit the node class (such as MyExtractNode) and build specific extract or load usage logic - In a specific node class (such as MyExtractNode), specify the corresponding Flink connector - Use specific node classes in specific ETL implementation logic (such as MyExtractNode) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
