This is an automated email from the ASF dual-hosted git repository.
dockerzhang pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/inlong-website.git
The following commit(s) were added to refs/heads/master by this push:
new b70dac8fe1 [INLONG-782][Doc][Elasticsearch] Describe tables using
MarkDown (#786)
b70dac8fe1 is described below
commit b70dac8fe15c993ce7c348ae2c572a3a57b31235
Author: emhui <[email protected]>
AuthorDate: Wed Jun 28 15:16:09 2023 +0800
[INLONG-782][Doc][Elasticsearch] Describe tables using MarkDown (#786)
---
docs/data_node/load_node/elasticsearch.md | 308 +++------------------
.../current/data_node/load_node/elasticsearch.md | 305 +++-----------------
2 files changed, 84 insertions(+), 529 deletions(-)
diff --git a/docs/data_node/load_node/elasticsearch.md
b/docs/data_node/load_node/elasticsearch.md
index 5a59ed248d..8308b5bcb6 100644
--- a/docs/data_node/load_node/elasticsearch.md
+++ b/docs/data_node/load_node/elasticsearch.md
@@ -74,190 +74,28 @@ TODO: It will be supported in the future.
## Elasticsearch Load Node Options
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style={{width: '25%'}}>Option</th>
- <th class="text-center" style={{width: '8%'}}>Required</th>
- <th class="text-center" style={{width: '7%'}}>Default</th>
- <th class="text-center" style={{width: '10%'}}>Type</th>
- <th class="text-center" style={{width: '50%'}}>Description</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>connector</h5></td>
- <td>required</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Specify what connector to use, valid values are:
- <ul>
- <li><code>elasticsearch-6-inlong</code>: connect to Elasticsearch 5.x
and 6.x cluster.</li>
- <li><code>elasticsearch-7-inlong</code>: connect to Elasticsearch 7.x
and later versions cluster.</li>
- </ul></td>
- </tr>
- <tr>
- <td><h5>hosts</h5></td>
- <td>required</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>One or more Elasticsearch hosts to connect to, e.g.
<code>'http://host_name:9092;http://host_name:9093'</code>.</td>
- </tr>
- <tr>
- <td><h5>index</h5></td>
- <td>required</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Elasticsearch index for every record. Can be a static index (e.g.
<code>'myIndex'</code>) or
- a dynamic index (e.g. <code>'index-\{'{log_ts|yyyy-MM-dd}'}'</code>).
- See the following <a href="#dynamic-index">Dynamic Index</a> section
for more details.</td>
- </tr>
- <tr>
- <td><h5>document-type</h5></td>
- <td>required in 5.x and 6.x</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Elasticsearch document type. Not necessary anymore in
<code>elasticsearch-7</code>.</td>
- </tr>
- <tr>
- <td><h5>document-id.key-delimiter</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>_</td>
- <td>String</td>
- <td>Delimiter for composite keys ("_" by default), e.g., "$" would
result in IDs "KEY1$KEY2$KEY3".</td>
- </tr>
- <tr>
- <td><h5>username</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Username used to connect to Elasticsearch instance. Please notice
that Elasticsearch doesn't pre-bundled security feature, but you can enable it
by following the <a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-security.html";>guideline</a>
to secure an Elasticsearch cluster.</td>
- </tr>
- <tr>
- <td><h5>password</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Password used to connect to Elasticsearch instance. If
<code>username</code> is configured, this option must be configured with
non-empty string as well.</td>
- </tr>
- <tr>
- <td><h5>failure-handler</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>fail</td>
- <td>String</td>
- <td>Failure handling strategy in case a request to Elasticsearch fails.
Valid strategies are:
- <ul>
- <li><code>fail</code>: throws an exception if a request fails and thus
causes a job failure.</li>
- <li><code>ignore</code>: ignores failures and drops the request.</li>
- <li><code>retry-rejected</code>: re-adds requests that have failed due
to queue capacity saturation.</li>
- <li>custom class name: for failure handling with a
ActionRequestFailureHandler subclass.</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td><h5>sink.flush-on-checkpoint</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>true</td>
- <td>Boolean</td>
- <td>Flush on checkpoint or not. When disabled, a sink will not wait for
all pending action requests
- to be acknowledged by Elasticsearch on checkpoints. Thus, a sink does
NOT provide any strong
- guarantees for at-least-once delivery of action requests.
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.max-actions</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>1000</td>
- <td>Integer</td>
- <td>Maximum number of buffered actions per bulk request.
- Can be set to <code>'0'</code> to disable it.
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.max-size</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>2mb</td>
- <td>MemorySize</td>
- <td>Maximum size in memory of buffered actions per bulk request. Must be
in MB granularity.
- Can be set to <code>'0'</code> to disable it.
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.interval</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>1s</td>
- <td>Duration</td>
- <td>The interval to flush buffered actions.
- Can be set to <code>'0'</code> to disable it. Note, both
<code>'sink.bulk-flush.max-size'</code> and
<code>'sink.bulk-flush.max-actions'</code>
- can be set to <code>'0'</code> with the flush interval set allowing
for complete async processing of buffered actions.
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.backoff.strategy</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>DISABLED</td>
- <td>String</td>
- <td>Specify how to perform retries if any flush actions failed due to a
temporary request error. Valid strategies are:
- <ul>
- <li><code>DISABLED</code>: no retry performed, i.e. fail after the
first request error.</li>
- <li><code>CONSTANT</code>: wait for backoff delay between retries.</li>
- <li><code>EXPONENTIAL</code>: initially wait for backoff delay and
increase exponentially between retries.</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.backoff.max-retries</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>8</td>
- <td>Integer</td>
- <td>Maximum number of backoff retries.</td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.backoff.delay</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>50ms</td>
- <td>Duration</td>
- <td>Delay between each backoff attempt. For <code>CONSTANT</code>
backoff, this is simply the delay between each retry. For
<code>EXPONENTIAL</code> backoff, this is the initial base delay.</td>
- </tr>
- <tr>
- <td><h5>connection.max-retry-timeout</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>Duration</td>
- <td>Maximum timeout between retries.</td>
- </tr>
- <tr>
- <td><h5>connection.path-prefix</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Prefix string to be added to every REST communication, e.g.,
<code>'/v1'</code>.</td>
- </tr>
- <tr>
- <td><h5>routing.filed-name</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Using field value in the record to dynamically generate routing
filed.</td>
- </tr>
- <tr>
- <td><h5>format</h5></td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>json</td>
- <td>String</td>
- <td>Elasticsearch connector supports to specify a format. The format
must produce a valid json document.
- By default uses built-in <code>'json'</code> format. Please refer to <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/formats/overview/";>JSON
Format</a> page for more details.
- </td>
- </tr>
- <tr>
- <td>inlong.metric.labels</td>
- <td>optional</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Inlong metric label, format of value is
groupId=[groupId]&streamId=[streamId]&nodeId=[nodeId].</td>
- </tr>
- </tbody>
-</table>
+| Option | Required | Default | Type | Description |
+| --- | --- | --- | --- | --- |
+| connector | required | (none) | String | Specify what connector to use,
valid values are:<br/><br/>- `elasticsearch-6-inlong`: connect to Elasticsearch
5.x and 6.x cluster.<br/>- `elasticsearch-7-inlong`: connect to Elasticsearch
7.x and later versions cluster. |
+| hosts | required | (none) | String | One or more Elasticsearch hosts to
connect to, e.g. `'http://host_name:9092;http://host_name:9093'`. |
+| index | required | (none) | String | Elasticsearch index for every record.
Can be a static index (e.g. `'myIndex'`) or a dynamic index (e.g.
`'index-\{log_ts\|yyyy-MM-dd}'`). See the following [Dynamic
Index](#dynamic-index) section for more details. |
+| document-type | required in 5.x and 6.x | (none) | String | Elasticsearch
document type. Not necessary anymore in `elasticsearch-7`. |
+| document-id.key-delimiter | optional | _ | String | Delimiter for
composite keys ("_" by default), e.g., "$" would result in IDs
"KEY1\$KEY2\$KEY3". |
+| username | optional | (none) | String | Username used to connect to
Elasticsearch instance. Please notice that Elasticsearch doesn't pre-bundled
security feature, but you can enable it by following the
[guideline](https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-security.html)
to secure an Elasticsearch cluster. |
+| password | optional | (none) | String | Password used to connect to
Elasticsearch instance. If `username` is configured, this option must be
configured with non-empty string as well. |
+| failure-handler | optional | fail | String | Failure handling strategy in
case a request to Elasticsearch fails. Valid strategies are:<br/><br/>- `fail`:
throws an exception if a request fails and thus causes a job failure.<br/>-
`ignore`: ignores failures and drops the request.<br/>- `retry-rejected`:
re-adds requests that have failed due to queue capacity saturation.<br/>-
custom class name: for failure handling with a ActionRequestFailureHandler
subclass. |
+| sink.flush-on-checkpoint | optional | true | Boolean | Flush on checkpoint
or not. When disabled, a sink will not wait for all pending action requests to
be acknowledged by Elasticsearch on checkpoints. Thus, a sink does NOT provide
any strong guarantees for at-least-once delivery of action requests. |
+| sink.bulk-flush.max-actions | optional | 1000 | Integer | Maximum number of
buffered actions per bulk request. Can be set to `'0'` to disable it. |
+| sink.bulk-flush.max-size | optional | 2mb | MemorySize | Maximum size in
memory of buffered actions per bulk request. Must be in MB granularity. Can be
set to `'0'` to disable it. |
+ sink.bulk-flush.interval | optional | 1s | Duration | The interval to flush
buffered actions. Can be set to `'0'` to disable it. Note, both
`'sink.bulk-flush.max-size'` and `'sink.bulk-flush.max-actions'`can be set to
`'0'` with the flush interval set allowing for complete async processing of
buffered actions. |
+| sink.bulk-flush.backoff.strategy | optional | DISABLED | String | Specify
how to perform retries if any flush actions failed due to a temporary request
error. Valid strategies are:<br/><br/>- `DISABLED`: no retry performed, i.e.
fail after the first request error.<br/>- `CONSTANT`: wait for backoff delay
between retries.<br/>- `EXPONENTIAL`: initially wait for backoff delay and
increase exponentially between retries. |
+| sink.bulk-flush.backoff.max-retries | optional | 8 | Integer | Maximum
number of backoff retries. |
+| sink.bulk-flush.backoff.delay | optional | 50ms | Duration | Delay between
each backoff attempt. For `CONSTANT` backoff, this is simply the delay between
each retry. For `EXPONENTIAL` backoff, this is the initial base delay. |
+| connection.max-retry-timeout | optional | (none) | Duration | Maximum
timeout between retries. |
+| connection.path-prefix | optional | (none) | String | Prefix string to be
added to every REST communication, e.g., `'/v1'`. |
+| routing.filed-name | optional | (none) | String | Using field value in the
record to dynamically generate routing filed. |
+| format | optional | json | String | Elasticsearch connector supports to
specify a format. The format must produce a valid json document. By default
uses built-in `'json'` format. Please refer to [JSON
Format](https://nightlies.apache.org/flink/flink-docs-release-1.13/docs/connectors/table/formats/overview/)
page for more details. |
+| inlong.metric.labels | optional | (none) | String | Inlong metric label,
format of value is groupId=[groupId]&streamId=[streamId]&nodeId=[nodeId]. |
Features
----------------
@@ -290,85 +128,23 @@ For example, if the option value is
`'myusers-{'{log_ts|yyyy-MM-dd}'}'`, then a
## Data Type Mapping
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left">JSON type</th>
- <th class="text-left">Flink SQL type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>string</code></td>
- <td><code>CHAR / VARCHAR / STRING</code></td>
- </tr>
- <tr>
- <td><code>boolean</code></td>
- <td><code>BOOLEAN</code></td>
- </tr>
- <tr>
- <td><code>string with encoding: base64</code></td>
- <td><code>BINARY / VARBINARY</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>DECIMAL</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>TINYINT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>SMALLINT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>INT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>BIGINT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>FLOAT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>DOUBLE</code></td>
- </tr>
- <tr>
- <td><code>string with format: date</code></td>
- <td><code>DATE</code></td>
- </tr>
- <tr>
- <td><code>string with format: time</code></td>
- <td><code>TIME</code></td>
- </tr>
- <tr>
- <td><code>string with format: date-time</code></td>
- <td><code>TIMESTAMP</code></td>
- </tr>
- <tr>
- <td><code>string with format: date-time (with UTC time zone)</code></td>
- <td><code>TIMESTAMP_WITH_LOCAL_TIME_ZONE</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>INTERVAL</code></td>
- </tr>
- <tr>
- <td><code>array</code></td>
- <td><code>ARRAY</code></td>
- </tr>
- <tr>
- <td><code>object</code></td>
- <td><code>MAP / MULTISET</code></td>
- </tr>
- <tr>
- <td><code>object</code></td>
- <td><code>ROW</code></td>
- </tr>
- </tbody>
-</table>
\ No newline at end of file
+| JSON type | Flink SQL type |
+| --- | --- |
+| string | CHAR / VARCHAR / STRING |
+| boolean | BOOLEAN |
+| string with encoding: base64 | BINARY / VARBINARY |
+| number | DECIMAL |
+| number | TINYINT |
+| number | SMALLINT |
+| number | INT |
+| number | BIGINT |
+| number | FLOAT |
+| number | DOUBLE |
+| string with format: date | DATE |
+| string with format: time | TIME |
+| string with format: date-time | TIMESTAMP |
+| string with format: date-time (with UTC time zone) |
TIMESTAMP_WITH_LOCAL_TIME_ZONE |
+| number | INTERVAL |
+| array | ARRAY |
+| object | MAP / MULTISET |
+| object | ROW |
\ No newline at end of file
diff --git
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md
index 838fe01600..e01248355c 100644
---
a/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md
+++
b/i18n/zh-CN/docusaurus-plugin-content-docs/current/data_node/load_node/elasticsearch.md
@@ -74,186 +74,28 @@ TODO: 将在未来支持这个特性。
## Elasticsearch Load 节点参数
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left" style={{width: '25%'}}>参数</th>
- <th class="text-center" style={{width: '8%'}}>是否必选</th>
- <th class="text-center" style={{width: '7%'}}>默认值</th>
- <th class="text-center" style={{width: '10%'}}>数据类型</th>
- <th class="text-center" style={{width: '50%'}}>描述</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><h5>connector</h5></td>
- <td>必选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>指定要使用的连接器，有效值为：
- <ul>
- <li><code>elasticsearch-6</code>：连接到 Elasticsearch 5.x and 6.x 的集群。</li>
- <li><code>elasticsearch-7</code>：连接到 Elasticsearch 7.x 及更高版本的集群。</li>
- </ul></td>
- </tr>
- <tr>
- <td><h5>hosts</h5></td>
- <td>必选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>要连接到的一台或多台 Elasticsearch 主机，例如
<code>'http://host_name:9092;http://host_name:9093'</code>。</td>
- </tr>
- <tr>
- <td><h5>index</h5></td>
- <td>必选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Elasticsearch 中每条记录的索引。可以是一个静态索引（例如
<code>'myIndex'</code>）或一个动态索引（例如 <code>'index-{'{log_ts|yyyy-MM-dd}'}'</code>）。
- 更多详细信息，请参见下面的<a href="#dymic-index">动态索引</a>部分。</td>
- </tr>
- <tr>
- <td><h5>document-type</h5></td>
- <td>6.x 版本中必选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>Elasticsearch 文档类型。在 <code>elasticsearch-7</code> 中不再需要。</td>
- </tr>
- <tr>
- <td><h5>document-id.key-delimiter</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>_</td>
- <td>String</td>
- <td>复合键的分隔符（默认为"_"），例如，指定为"$"将导致文档 ID 为"KEY1$KEY2$KEY3"。</td>
- </tr>
- <tr>
- <td><h5>username</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>用于连接 Elasticsearch 实例的用户名。请注意，Elasticsearch 没有预绑定安全特性，但你可以通过如下<a
href="https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-security.html";>指南</a>启用它来保护
Elasticsearch 集群。</td>
- </tr>
- <tr>
- <td><h5>password</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>用于连接 Elasticsearch
实例的密码。如果配置了<code>username</code>，则此选项也必须配置为非空字符串。</td>
- </tr>
- <tr>
- <td><h5>failure-handler</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>fail</td>
- <td>String</td>
- <td>对 Elasticsearch 请求失败情况下的失败处理策略。有效策略为：
- <ul>
- <li><code>fail</code>：如果请求失败并因此导致作业失败，则抛出异常。</li>
- <li><code>ignore</code>：忽略失败并放弃请求。</li>
- <li><code>retry-rejected</code>：重新添加由于队列容量饱和而失败的请求。</li>
- <li>自定义类名称：使用 ActionRequestFailureHandler 的子类进行失败处理。</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td><h5>sink.flush-on-checkpoint</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>true</td>
- <td>Boolean</td>
- <td>是否在 checkpoint 时执行 flush。禁用后，在 checkpoint 时 sink 将不会等待所有的 pending
请求被 Elasticsearch 确认。因此，sink 不会为请求的 at-least-once 交付提供任何有力保证。
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.max-actions</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>1000</td>
- <td>Integer</td>
- <td>每个批量请求的最大缓冲操作数。
- 可以设置为<code>'0'</code>来禁用它。
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.max-size</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>2mb</td>
- <td>MemorySize</td>
- <td>每个批量请求的缓冲操作在内存中的最大值。单位必须为 MB。
- 可以设置为<code>'0'</code>来禁用它。
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.interval</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>1s</td>
- <td>Duration</td>
- <td>flush 缓冲操作的间隔。
-
可以设置为<code>'0'</code>来禁用它。注意，<code>'sink.bulk-flush.max-size'</code>和<code>'sink.bulk-flush.max-actions'</code>都设置为<code>'0'</code>的这种
flush 间隔设置允许对缓冲操作进行完全异步处理。
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.backoff.strategy</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>DISABLED</td>
- <td>String</td>
- <td>指定在由于临时请求错误导致任何 flush 操作失败时如何执行重试。有效策略为：
- <ul>
- <li><code>DISABLED</code>：不执行重试，即第一次请求错误后失败。</li>
- <li><code>CONSTANT</code>：等待重试之间的回退延迟。</li>
- <li><code>EXPONENTIAL</code>：先等待回退延迟，然后在重试之间指数递增。</li>
- </ul>
- </td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.backoff.max-retries</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>8</td>
- <td>Integer</td>
- <td>最大回退重试次数。</td>
- </tr>
- <tr>
- <td><h5>sink.bulk-flush.backoff.delay</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>50ms</td>
- <td>Duration</td>
- <td>每次回退尝试之间的延迟。对于 <code>CONSTANT</code> 回退策略，该值是每次重试之间的延迟。对于
<code>EXPONENTIAL</code> 回退策略，该值是初始的延迟。</td>
- </tr>
- <tr>
- <td><h5>connection.max-retry-timeout</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>Duration</td>
- <td>最大重试超时时间。</td>
- </tr>
- <tr>
- <td><h5>connection.path-prefix</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>添加到每个 REST 通信中的前缀字符串，例如，<code>'/v1'</code>。</td>
- </tr>
- <tr>
- <td><h5>routing.filed-name</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>使用 field 值来生成该 field 的动态路由</td>
- </tr>
- <tr>
- <td><h5>format</h5></td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>json</td>
- <td>String</td>
- <td>Elasticsearch 连接器支持指定格式。该格式必须生成一个有效的 json 文档。
- 默认使用内置的 <code>'json'</code> 格式。更多详细信息，请参阅 <a
href="https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/formats/overview/";>JSON
Format</a> 页面。
- </td>
- </tr>
- <tr>
- <td>inlong.metric.labels</td>
- <td>可选</td>
- <td style={{wordWrap: 'break-word'}}>(none)</td>
- <td>String</td>
- <td>inlong metric
的标签值，该值的构成为groupId=[groupId]&streamId=[streamId]&nodeId=[nodeId]。</td>
- </tr>
- </tbody>
-</table>
+| 参数 | 是否必选 | 默认值 | 数据类型 | 描述
|
+| --- | --- | --- | ---
|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| connector | 必选 | (none) | String | 指定要使用的连接器，有效值为：<br/><br/>-
`elasticsearch-6`：连接到 Elasticsearch 5.x and 6.x 的集群。<br/>-
`elasticsearch-7`：连接到 Elasticsearch 7.x 及更高版本的集群。
|
+| hosts | 必选 | (none) | String | 要连接到的一台或多台 Elasticsearch 主机，例如
`'http://host_name:9092;http://host_name:9093'`。
|
+| index | 必选 | (none) | String | Elasticsearch 中每条记录的索引。可以是一个静态索引（例如
`'myIndex'`）或一个动态索引（例如 `'index-{log_ts\|yyyy-MM-dd}'`）。
更多详细信息，请参见下面的[动态索引](#动态索引)部分。
|
+| document-type | 6.x 版本中必选 | (none) | String | Elasticsearch 文档类型。在
`elasticsearch-7` 中不再需要。
|
+| document-id.key-delimiter | 可选 | _ | String |
复合键的分隔符（默认为"_"），例如，指定为"$"将导致文档 ID 为"KEY1\$KEY2\$KEY3"。
|
+| username | 可选 | (none) | String | 用于连接 Elasticsearch
实例的用户名。请注意，Elasticsearch
没有预绑定安全特性，但你可以通过如下[指南](https://www.elastic.co/guide/en/elasticsearch/reference/master/configuring-security.html)启用它来保护
Elasticsearch 集群。 |
+| password | 可选 | (none) | String | 用于连接 Elasticsearch
实例的密码。如果配置了`username`，则此选项也必须配置为非空字符串。
|
+| failure-handler | 可选 | fail | String | 对 Elasticsearch
请求失败情况下的失败处理策略。有效策略为：<br/><br/>- `fail`：如果请求失败并因此导致作业失败，则抛出异常。<br/>-
`ignore`：忽略失败并放弃请求。<br/>- `retry-rejected`：重新添加由于队列容量饱和而失败的请求。<br/>- 自定义类名称：使用
ActionRequestFailureHandler 的子类进行失败处理。 |
+| sink.flush-on-checkpoint | 可选 | true | Boolean | 是否在 checkpoint 时执行
flush。禁用后，在 checkpoint 时 sink 将不会等待所有的 pending 请求被 Elasticsearch 确认。因此，sink
不会为请求的 at-least-once 交付提供任何有力保证。
|
+| sink.bulk-flush.max-actions | 可选 | 1000 | Integer | 每个批量请求的最大缓冲操作数。
可以设置为`'0'`来禁用它。
|
+| sink.bulk-flush.max-size | 可选 | 2mb | MemorySize |
每个批量请求的缓冲操作在内存中的最大值。单位必须为 MB。 可以设置为`'0'`来禁用它。
|
+| sink.bulk-flush.interval | 可选 | 1s | Duration | flush 缓冲操作的间隔。
可以设置为`'0'`来禁用它。注意，`'sink.bulk-flush.max-size'`和`'sink.bulk-flush.max-actions'`都设置为`'0'`的这种
flush 间隔设置允许对缓冲操作进行完全异步处理。
|
+| sink.bulk-flush.backoff.strategy | 可选 | DISABLED | String |
指定在由于临时请求错误导致任何 flush 操作失败时如何执行重试。有效策略为：<br/><br/>-
`DISABLED`：不执行重试，即第一次请求错误后失败。<br/>- `CONSTANT`：等待重试之间的回退延迟。<br/>-
`EXPONENTIAL`：先等待回退延迟，然后在重试之间指数递增。
|
+| sink.bulk-flush.backoff.max-retries | 可选 | 8 | Integer | 最大回退重试次数。
|
+| sink.bulk-flush.backoff.delay | 可选 | 50ms | Duration | 每次回退尝试之间的延迟。对于
`CONSTANT` 回退策略，该值是每次重试之间的延迟。对于 `EXPONENTIAL` 回退策略，该值是初始的延迟。
|
+| connection.max-retry-timeout | 可选 | (none) | Duration | 最大重试超时时间。
|
+| connection.path-prefix | 可选 | (none) | String | 添加到每个 REST
通信中的前缀字符串，例如，`'/v1'`。
|
+| routing.filed-name | 可选 | (none) | String | 使用 field 值来生成该 field 的动态路由
|
+| format | 可选 | json | String | Elasticsearch 连接器支持指定格式。该格式必须生成一个有效的 json
文档。 默认使用内置的 `'json'` 格式。更多详细信息，请参阅 [JSON
Format](https://nightlies.apache.org/flink/flink-docs-release-1.13/zh/docs/connectors/table/formats/overview/)
页面。 |
+| inlong.metric.labels | 可选 | (none) | String | inlong metric 的标签值，该值的构成为
groupId=[groupId]&streamId=[streamId]&nodeId=[nodeId]。
|
特性
----------------
@@ -285,86 +127,23 @@ Elasticsearch Load 节点同时支持静态索引和动态索引。
例如，如果选项值设置为 `'myusers-{'{log_ts|yyyy-MM-dd}'}'`，则 `log_ts` 字段值为 `2020-03-27
12:25:55` 的记录将被写入到 "myusers-2020-03-27" 索引中。
## 数据类型映射
-
-<table class="table table-bordered">
- <thead>
- <tr>
- <th class="text-left">JSON type</th>
- <th class="text-left">Flink SQL type</th>
- </tr>
- </thead>
- <tbody>
- <tr>
- <td><code>string</code></td>
- <td><code>CHAR / VARCHAR / STRING</code></td>
- </tr>
- <tr>
- <td><code>boolean</code></td>
- <td><code>BOOLEAN</code></td>
- </tr>
- <tr>
- <td><code>string with encoding: base64</code></td>
- <td><code>BINARY / VARBINARY</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>DECIMAL</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>TINYINT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>SMALLINT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>INT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>BIGINT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>FLOAT</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>DOUBLE</code></td>
- </tr>
- <tr>
- <td><code>string with format: date</code></td>
- <td><code>DATE</code></td>
- </tr>
- <tr>
- <td><code>string with format: time</code></td>
- <td><code>TIME</code></td>
- </tr>
- <tr>
- <td><code>string with format: date-time</code></td>
- <td><code>TIMESTAMP</code></td>
- </tr>
- <tr>
- <td><code>string with format: date-time (with UTC time zone)</code></td>
- <td><code>TIMESTAMP_WITH_LOCAL_TIME_ZONE</code></td>
- </tr>
- <tr>
- <td><code>number</code></td>
- <td><code>INTERVAL</code></td>
- </tr>
- <tr>
- <td><code>array</code></td>
- <td><code>ARRAY</code></td>
- </tr>
- <tr>
- <td><code>object</code></td>
- <td><code>MAP / MULTISET</code></td>
- </tr>
- <tr>
- <td><code>object</code></td>
- <td><code>ROW</code></td>
- </tr>
- </tbody>
-</table>
\ No newline at end of file
+| JSON type | Flink SQL type |
+| --- | --- |
+| string | CHAR / VARCHAR / STRING |
+| boolean | BOOLEAN |
+| string with encoding: base64 | BINARY / VARBINARY |
+| number | DECIMAL |
+| number | TINYINT |
+| number | SMALLINT |
+| number | INT |
+| number | BIGINT |
+| number | FLOAT |
+| number | DOUBLE |
+| string with format: date | DATE |
+| string with format: time | TIME |
+| string with format: date-time | TIMESTAMP |
+| string with format: date-time (with UTC time zone) |
TIMESTAMP_WITH_LOCAL_TIME_ZONE |
+| number | INTERVAL |
+| array | ARRAY |
+| object | MAP / MULTISET |
+| object | ROW |
\ No newline at end of file
