This is an automated email from the ASF dual-hosted git repository.
xiatian pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/eventmesh-site.git
The following commit(s) were added to refs/heads/master by this push:
new 17eb85c6f [ISSUE #220] Add HTTP Sink Connector Document (#221)
17eb85c6f is described below
commit 17eb85c6f03541c215f6c82c1b8feac400e6d944
Author: Zaki <[email protected]>
AuthorDate: Sat May 11 20:51:09 2024 +0800
[ISSUE #220] Add HTTP Sink Connector Document (#221)
* docs: add HTTP Sink Connector Chinese docs
* docs: add HTTP Sink Connector English docs
* fix:update something
* fix:update something
---
.../03-connect/02-http-connector.md | 167 +++++++++++++++++++--
docs/roadmap.md | 2 +-
.../03-connect/02-http-connector.md | 141 ++++++++++++++++-
3 files changed, 294 insertions(+), 16 deletions(-)
diff --git a/docs/design-document/03-connect/02-http-connector.md
b/docs/design-document/03-connect/02-http-connector.md
index 3aeeb5b82..b62b77695 100644
--- a/docs/design-document/03-connect/02-http-connector.md
+++ b/docs/design-document/03-connect/02-http-connector.md
@@ -5,12 +5,14 @@
### Configuration
Before using HTTP source connector, you need to configure the server.
+
- Please configure `sourceEnable` to `true` in `/resource/server-config.yml`
to enable source functionality.
-- Please configure the source connector in `/resource/source-config.yml`, only
the configuration under `connectorConfig` is described here:
- - `connectorName`, name of the connector.
- - (required) `path`, path of the API.
- - (required) `port`, port of the API.
- - `idleTimeout`, idle TCP connection timeout in seconds. A connection will
timeout and be closed if no data is received nor sent within the `idleTimeout`
seconds. The default is 0, which means don't timeout.
+- Please configure the source connector in `/resource/source-config.yml`, only
the configuration under `connectorConfig`
+ is described here:
+ - `connectorName`, name of the connector.
+ - (required) `path`, path of the API.
+ - (required) `port`, port of the API.
+ - `idleTimeout`, idle TCP connection timeout in seconds. A connection will
timeout and be closed if no data is received nor sent within the `idleTimeout`
seconds. The default is 0, which means don't timeout.
### Startup
@@ -25,15 +27,16 @@ You can send messages to the source connector via HTTP.
```yaml
connectorConfig:
- connectorName: httpSource
- path: /test
- port: 3755
- idleTimeout: 5
+ connectorName: httpSource
+ path: /test
+ port: 3755
+ idleTimeout: 5
```
The above example configures a URL `http://localhost:3755/test` in
`source-config.yml`.
-You can send messages in `binary` mode or `structured` mode as specified in
[cloudevent-spec](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/http-protocol-binding.md).
+You can send messages in `binary` mode or `structured` mode as specified
+in
[cloudevent-spec](https://github.com/cloudevents/spec/blob/v1.0.2/cloudevents/bindings/http-protocol-binding.md).
Here are two examples:
@@ -65,3 +68,147 @@ curl --location --request POST 'http://localhost:3755/test'
\
"data": "testdata"
}'
```
+
+## HTTP Sink Connector
+
+The HTTP sink connector has two modes: common and webhook.
+
+First of all, either mode has the basic function of sinkConnector, which is to
send a message to the target HTTP server.
+When the response is received from the HTTP server, the common mode only cares
about the status code of the response to
+determine whether it was sent successfully, while the webook mode not only
cares about whether it was sent successfully
+or not, but also stores the data of the response (called the callback data)
and provides callback data exposure services
+to the outside world.
+
+The specific differences between the two are as follows:
+
+- Default Idle TCP Connection Timeout
+
+ The default value is 5000ms for common mode and 15000ms for webhook mode.
+
+- Processing of response results
+
+ The common mode only cares about the status code of the response, while the
webhook mode will also store the callback
+ data and provide callback data exposure services to the public.
+
+### Configuration
+
+Before using the HTTP sink connector, you need to configure the sink.
+
+- Please configure `sinkEnable` to true in `/resource/server-config.yml` to
enable sink function.
+
+- Please configure the sink connector in `/resource/sink-config.yml`, only the
configuration under `connectorConfig` is
+ described here:
+
+ The simplest configuration of the common mode:
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ ```
+
+ The simplest configuration for webhook mode:
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ webhookConfig:
+ activate: true
+ port: 8988
+ ```
+
+All configurations for the HTTP sink connector are listed below:
+
+| Field | Type | Required or not |
Clarification | Remark
|
+|---------------------------------|---------|--------------------|------------------------------------------------------------|----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connectorName | String | yes | Connector
Name |
|
+| urls | List | yes | List of
URLs to receive messages | The default port is 80 for
HTTP protocol and 443 for HTTPS protocol<br />The legal URL format are:<br
/>http://127.0.0.1:8987/test<br />http://127.0.0.1/test<br
/>https://example.com:4943/test<br/>https://example.com/test<br/>
|
+| keepAlive | Boolean | no | Whether to
use HTTP persistent connection | Default: true
|
+| keepAliveTimeout | Integer | no | HTTP
persistent connection timeout | Unit: ms,default: 60000
|
+| connectionTimeout | Integer | no | TCP
connection timeout | Unit: ms,default: 5000
|
+| idleTimeout | Integer | no | TCP idle
timeout | Unit: ms,default:
5000(common), 15000(webhook)
|
+| maxConnectionPoolSize | Integer | no | Maximum
number of HTTP connections for the client | Default: 5
|
+| retryConfig | Object | |
Configuration of the retry mechanism |
|
+| retryConfig.maxRetries | Integer | no | Maximum
number of retries | Does not include first
attempt, default: 2
|
+| retryConfig.interval | Integer | no | Retry
interval | Unit: ms, default: 2000
|
+| retryConfig.retryOnNonSuccess | Boolean | no | Whether to
retry requests that receive a non-2xx response | Default: false, retries only
network level error requests
|
+| webhookConfig | Object | |
Configuration of Webhook Mode |
|
+| webhookConfig.activate | Boolean | no | Whether to
enable Webhook mode | Default: false
|
+| webhookConfig.exportPath | String | no | Callback
data exposure path | Default: /export<br/>The
request method is fixed to GET<br/>Request Parameters:<br />type：peek (default
value, get data)/poll (get and delete data)<br />pageNum: must be greater than
0 when type is peek, invalid when type is poll<br/>pageSize: must be greater
than 0 |
+| webhookConfig.port | Integer | Yes (when enabled) | Callback
data exposure port |
|
+| webhookConfig.serverIdleTimeout | Integer | no | TCP idle
timeout length for servers exposing callback data | Unit: ms, default: 5000
|
+| webhookConfig.maxStorageSize | Integer | no | Store the
maximum value of the callback data queue | Default: 5000
|
+
+### Startup
+
+1. start EventMesh Runtime
+2. start eventmesh-connector-http
+
+Once that's done, send a message to the EventMesh, and then your message will
happen to the target HTTP server via an
+HTTP request. If webhook mode is enabled, an HTTP server will also be opened
at the same time to provide callback data
+exposure to the outside world.
+
+### Data Format
+
+The data format of the message sent by the HTTP sink connector is as follows:
+
+- type: consists of connectorName, protocol, and mode.
+- time: the time the request was sent
+- uuid: request unique identifier, one-to-one with the uuid of the callback
data structure in webhook mode
+- eventId: event ID, composed of type and offset
+- data：actual data to be sent
+
+```json
+{
+ "type": "httpSink.http.webhook",
+ "time": "2024-05-09T22:27:02.028",
+ "uuid": "951d88ee-2503-4df3-a1dd-6e4a6a1c3d3b",
+ "eventId": "httpSink.http.webhook-0",
+ "data": ${ConnectRecord}
+}
+```
+
+When webhook mode is enabled, an HTTP server is also enabled to provide
callback data exposure to the outside world. The
+format of the callback data is as follows:
+
+- pageNum: current page number
+- pageSize: size per page
+- pageItems: the callback data array obtained
+- The meaning of the fields of pageItem is as follows:
+ - data: callback data returned from the callback server
+ - metadata: metadata, the fields have the following meanings:
+ - url: the URL to get the callback data
+ - code: the status code of the response to the request callback data,
or -1 if a network level error occurred (
+ response could not be received).
+ - message: response information from the request callback data, or
exception information if a network level
+ error occurred (response could not be received).
+ - uuid: a unique request identifier for the callback data,
corresponding to the uuid carried in the message sent
+ above.
+ - receivedTime: response time for requesting callback data
+ - retryNum: number of retries
+ - retriedBy: record the uuid of the request being retried if a retry
is sent, otherwise null
+
+```json
+{
+ "pageNum": 1,
+ "pageSize": 10,
+ "pageItems": [
+ {
+ "data": "callbackData",
+ "metadata": {
+ "url": "http://127.0.0.1:8987/test";,
+ "code": 200,
+ "message": "OK",
+ "receivedTime": "2024-05-09 22:53:21.556",
+ "uuid": "fba29061-1a1f-4482-9c83-43ba4e0bcf3f",
+ "retriedBy": null,
+ "retryNum": 0
+ }
+ }
+ ]
+}
+```
+
diff --git a/docs/roadmap.md b/docs/roadmap.md
index 6fb92e334..9ac2e40b4 100644
--- a/docs/roadmap.md
+++ b/docs/roadmap.md
@@ -57,7 +57,7 @@ The development roadmap of Apache EventMesh is an overview of
the planned featur
|
[Kafka](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-kafka)
| ✅ | ✅ |
|
[Pulsar](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-pulsar)
| ✅ | ✅ |
| [RabbitMQ](./design-document/connect/rabbitmq-connector) | ✅ |
✅ |
-| [HTTP](./design-document/connect/http-connector) | ✅ |
⬜ |
+| [HTTP](./design-document/connect/http-connector) | ✅ |
✅ |
|
[JDBC](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-jdbc)
| ⬜ | ✅ |
|
[Spring](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-spring)
| ✅ | ✅ |
|
[OpenFunction](https://github.com/apache/eventmesh/tree/master/eventmesh-connectors/eventmesh-connector-openfunction)
| ✅ | ✅ |
diff --git
a/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
b/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
index 95287d5c9..710e47b8b 100644
---
a/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
+++
b/i18n/zh/docusaurus-plugin-content-docs/current/design-document/03-connect/02-http-connector.md
@@ -25,10 +25,10 @@
```yaml
connectorConfig:
- connectorName: httpSource
- path: /test
- port: 3755
- idleTimeout: 5
+ connectorName: httpSource
+ path: /test
+ port: 3755
+ idleTimeout: 5
```
上述的例子在`source-config.yml`中配置了一个 URL `http://localhost:3755/test`.
@@ -64,4 +64,135 @@ curl --location --request POST 'http://localhost:3755/test'
\
"datacontenttype":"text/plain",
"data": "testdata"
}'
-```
\ No newline at end of file
+```
+
+## HTTP Sink Connector
+
+HTTP sink connector 拥有两种模式：common 和 webhook。
+
+首先，无论是哪种模式，都具备sinkConnector的基本功能，即将消息发送给目标HTTP服务器。当收到HTTP服务器的响应时，common模式只关心响应的状态码，以判断是否发送成功，而webook模式除了关心是否发送成功以外，还将响应的数据（称为回调数据）进行存储，并对外提供回调数据暴露服务。两者的具体差别如下：
+
+- 默认空闲 TCP 连接超时时间
+
+ common模式的默认值为5000ms，webhook模式则为15000ms。
+
+- 响应结果的处理
+
+ common模式只关心响应的状态码，而webhook模式还会存储回调数据，并对外提供回调数据暴露服务。
+
+### 配置
+
+使用 HTTP sink connector 前，需要进行 sink 的配置。
+
+- 请在 `/resource/server-config.yml` 中配置 `sinkEnable`为`true` 以开启sink 功能。
+
+- 请在 `/resource/sink-config.yml`中配置 sink connector,在此仅说明 `connectorConfig`
下的配置：
+
+ common模式的最简配置：
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ ```
+
+ webhook模式的最简配置：
+
+ ```yaml
+ connectorConfig:
+ connectorName: httpSink
+ urls:
+ - http://127.0.0.1:8987/test
+ webhookConfig:
+ activate: true
+ port: 8988
+ ```
+
+HTTP sink connector 的所有配置如下：
+
+| 属性 | 类型 | 是否必填 | 说明 |
备注
|
+|---------------------------------|---------|--------|----------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connectorName | String | 是 | connector名称 |
|
+| urls | List | 是 | 接收消息的URL列表 |
HTTP协议默认端口为80,HTTPS协议则为443<br />合法URL格式为：<br />http://127.0.0.1:8987/test<br
/>http://127.0.0.1/test<br
/>https://example.com:4943/test<br/>https://example.com/test<br/> |
+| keepAlive | Boolean | 否 | 是否使用HTTP持久连接 |
默认：true
|
+| keepAliveTimeout | Integer | 否 | HTTP持久连接超时时长 |
单位：ms，默认：60000
|
+| connectionTimeout | Integer | 否 | TCP连接超时时长 |
单位：ms，默认：5000
|
+| idleTimeout | Integer | 否 | TCP空闲超时时长 |
单位：ms，默认：5000（common），15000（webhook）
|
+| maxConnectionPoolSize | Integer | 否 | 客户端的最大HTTP连接数 |
默认：5
|
+| retryConfig | Object | | 重试机制的相关配置 |
|
+| retryConfig.maxRetries | Integer | 否 | 最大重试次数 |
不包含首次尝试，默认：2
|
+| retryConfig.interval | Integer | 否 | 重试间隔 |
单位：ms，默认：2000
|
+| retryConfig.retryOnNonSuccess | Boolean | 否 | 是否重试收到非2xx响应的请求 |
默认：false，仅重试网络层面的错误请求
|
+| webhookConfig | Object | | Webhook模式的相关配置 |
|
+| webhookConfig.activate | Boolean | 否 | 是否启用Webhook模式 |
默认：false
|
+| webhookConfig.exportPath | String | 否 | 回调数据暴露路径 |
默认：/export<br/>请求方式固定为GET<br/>请求参数有<br />type：peek(默认值，获取数据)/poll(获取并删除数据)<br
/>pageNum：当type为peek时，必须大于0,当type为poll时，该参数无效<br/>pageSize：必须大于0
|
+| webhookConfig.port | Integer | 是（启用时） | 回调数据暴露端口 |
|
+| webhookConfig.serverIdleTimeout | Integer | 否 | 暴露回调数据的服务器的TCP空闲超时时长 |
单位：ms，默认：5000
|
+| webhookConfig.maxStorageSize | Integer | 否 | 存储回调数据队列的最大值 |
默认：5000
|
+
+### 启动
+
+1. 启动 EventMesh Runtime
+2. 启动 eventmesh-connector-http
+
+完成之后，向 EventMesh 发送消息，然后你的消息就会通过HTTP请求发生给目标HTTP服务器了。如果启用webhook模式，则还会同时开启一个
+HTTP 服务器对外提供回调数据暴露服务。
+
+### 数据格式
+
+HTTP sink connector 发送消息的数据格式如下：
+
+- type：由connectorName、协议、模式所组成
+- time：发送请求的时间
+- uuid：请求唯一标识，在webhook模式中和回调数据结构的uuid一一对应
+- eventId：事件ID，由type和offset组成
+- data：实际需要发送的数据
+
+```json
+{
+ "type": "httpSink.http.webhook",
+ "time": "2024-05-09T22:27:02.028",
+ "uuid": "951d88ee-2503-4df3-a1dd-6e4a6a1c3d3b",
+ "eventId": "httpSink.http.webhook-0",
+ "data": ${ConnectRecord}
+}
+```
+
+启用webhook模式时，则还会同时开启一个 HTTP 服务器对外提供回调数据暴露服务。获取的回调数据格式如下：
+
+- pageNum：当前页数
+- pageSize：每页的大小
+- pageItems：获取的回调数据数组
+- pageItem的字段含义如下：
+ - data：从回调服务器返回的回调数据
+ - metadata：元数据，字段含义如下：
+ - url: 获取回调数据的URL
+ - code：请求回调数据得到的响应状态码，如果发生的网络层面的错误（无法收到响应），则为-1
+ - message：请求回调数据得到的响应信息，如果发生的网络层面的错误（无法收到响应），则为相关异常信息
+ - uuid：请求回调数据的唯一请求标识，和上面发送消息时所携带的uuid一一对应
+ - receivedTime：请求回调数据得到的响应时间
+ - retryNum：重试次数
+ - retriedBy：如果发送重试，则记录被重试的请求的uuid，否则为null
+
+```json
+{
+ "pageNum": 1,
+ "pageSize": 10,
+ "pageItems": [
+ {
+ "data": "callbackData",
+ "metadata": {
+ "url": "http://127.0.0.1:8987/test";,
+ "code": 200,
+ "message": "OK",
+ "receivedTime": "2024-05-09 22:53:21.556",
+ "uuid": "fba29061-1a1f-4482-9c83-43ba4e0bcf3f",
+ "retriedBy": null,
+ "retryNum": 0
+ }
+ }
+ ]
+}
+```
+
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
