pandaapo commented on code in PR #150: URL: https://github.com/apache/eventmesh-site/pull/150#discussion_r1427554388
########## docs/design-document/03-connect/00-connectors.md: ########## @@ -2,22 +2,50 @@ ## Connector -A connector is a bridge that interacts with a specific external service or underlying data source (e.g., Databases) on behalf of user applications. A connector is either a Source or a Sink. +A connector is an image or instance that interacts with a specific external service or underlying data source (e.g., Databases) on behalf of user applications. A connector is either a Source or a Sink. ## Source A source connector obtains data from an underlying data producer, and delivers it to targets after original data has been transformed into CloudEvents. It doesn't limit the way how a source retrieves data. (e.g., A source may pull data from a message queue or act as an HTTP server waiting for data sent to it). -CloudEvents is a specification for describing event data in common formats to provide interoperability across services, platforms and systems. - ## Sink A sink connector receives CloudEvents and does some specific business logics. (e.g., A MySQL Sink extracts useful data from CloudEvents and writes them to a MySQL database). +## CloudEvents + +A specification for describing event data in common formats to provide interoperability across services, platforms and systems. + ## Implements Add a new connector by implementing the source/sink interface using [eventmesh-openconnect-java](https://github.com/apache/eventmesh/tree/master/eventmesh-openconnect/eventmesh-openconnect-java). +## Technical Solution + +### Structure and process + + +### Design Detail + + +### Describe + +#### Worker + +Worker is divided into Source Worker and Sink Worker, which are triggered by the `Application` class and implement the methods of the `ConnectorWorker` interface respectively, which include the worker's running life cycle, and the worker carries the running of the connector. Workers can be lightweight and independent through mirroring Running, the eventmesh-sdk-java module is integrated internally, and the cloudevents protocol is used to interact with eventmesh. Currently, the tcp client is used by default. In the future, support for dynamic configuration can be considered + +#### Connector + +Connectors are divided into Source Connector and Sink Connector. Connectors have their own configuration files and run independently. Workers perform reflective loading and configuration analysis to complete Connector initialization and subsequent operation. Source Connector implements the poll method, and Sink Connector implements The put method uniformly uses `ConnectorRecord` to carry data. Both Source Connector and Sink Connector can operate independently. + +#### ConnectorRecord with CloudEvents + +`ConnectorRecord` is a connector layer data protocol. When workers interact with eventmesh, a protocol adapter needs to be developed to convert `ConnectorRecord` to CloudEvents protocol. + +#### Registry + +The Registry module is responsible for storing the synchronization progress of synchronizing data of different Connector instances, ensuring high availability between multiple Connector images or instances. Review Comment: Which part of the codes does this correspond to? Or is this a plan for the future of Connector? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
