This is an automated email from the ASF dual-hosted git repository.
zhangshenghang pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new e284fbc258 [Docs][Connector-V2] Improve SLS connector docs (#11288)
e284fbc258 is described below
commit e284fbc25800b042203f3cc0caf43092a79a591e
Author: Daniel Carter <[email protected]>
AuthorDate: Fri Jul 3 08:21:16 2026 +0800
[Docs][Connector-V2] Improve SLS connector docs (#11288)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Sls.md | 60 ++++++++++++-----------
docs/en/connectors/source/Sls.md | 94 +++++++++++++++++++++++++-----------
docs/zh/connectors/sink/Sls.md | 75 ++++++++++++++--------------
docs/zh/connectors/source/Sls.md | 102 ++++++++++++++++++++++++++-------------
4 files changed, 209 insertions(+), 122 deletions(-)
diff --git a/docs/en/connectors/sink/Sls.md b/docs/en/connectors/sink/Sls.md
index 757aa446c7..2510f94cb7 100644
--- a/docs/en/connectors/sink/Sls.md
+++ b/docs/en/connectors/sink/Sls.md
@@ -8,7 +8,7 @@ import ChangeLog from '../changelog/connector-sls.md';
> Spark<br/>
> Flink<br/>
-> Seatunnel Zeta<br/>
+> SeaTunnel Zeta<br/>
## Key Features
@@ -18,44 +18,47 @@ import ChangeLog from '../changelog/connector-sls.md';
## Description
-Sink connector for Aliyun Sls.
+The Sls sink connector writes SeaTunnel rows to Alibaba Cloud Simple Log
Service (SLS). Each
+SeaTunnel row is serialized as JSON and written to SLS as a log item whose
content key is `content`.
## Supported DataSource Info
-In order to use the Sls connector, the following dependencies are required.
-They can be downloaded via install-plugin.sh or from the Maven central
repository.
+To use the Sls connector, download the following dependency by using
`install-plugin.sh` or from the
+Maven central repository.
| Datasource | Supported Versions | Maven
|
|------------|--------------------|-----------------------------------------------------------------------------------|
| Sls | Universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls)
|
-## Source Options
+## Sink Options
-| Name | Type | Required | Default
| Description
|
-|-------------------------------------|---------|----------|------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
-| project | String | Yes | -
| [Aliyun Sls
Project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project?spm=a2c4g.11186623.0.0.6f9755ebyfaYSl)
|
-| logstore | String | Yes | -
| [Aliyun Sls
Logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore?spm=a2c4g.11186623.0.0.13137c08nfuiBC)
|
-| endpoint | String | Yes | -
| [Aliyun Access
Endpoint](https://help.aliyun.com/zh/sls/developer-reference/api-sls-2020-12-30-endpoint?spm=a2c4g.11186623.0.0.548945a8UyJULa)
|
-| access_key_id | String | Yes | -
| [Aliyun AccessKey
ID](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| access_key_secret | String | Yes | -
| [Aliyun AccessKey
Secret](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| source | String | No | SeaTunnel-Source
| Data Source marking in sls
|
-| topic | String | No | SeaTunnel-Topic
| Data topic marking in sls
|
+| Name | Type | Required | Default | Description
|
+|-------------------|--------|----------|--------------------|---------------------------------------------------------------------------------------------------------|
+| endpoint | String | Yes | - | Alibaba Cloud
SLS endpoint, for example `cn-hangzhou.log.aliyuncs.com` or an intranet
endpoint. |
+| project | String | Yes | - | [Alibaba Cloud
SLS project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project).
|
+| logstore | String | Yes | - | [Alibaba Cloud
SLS logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore).
|
+| access_key_id | String | Yes | - | Alibaba Cloud
AccessKey ID.
|
+| access_key_secret | String | Yes | - | Alibaba Cloud
AccessKey secret.
|
+| source | String | No | `SeaTunnel-Source` | Source tag
written to SLS log groups.
|
+| topic | String | No | `SeaTunnel-Topic` | Topic tag
written to SLS log groups.
|
+| log_group_size | Int | No | 100 | Configured SLS
log group write size. The current writer sends one SeaTunnel row in each write
request. |
-## Task Example
+## Notes
-### Simple
+- The configured RAM user must have permission to write logs to the target
project and logstore.
+- The sink writes data as soon as `write` is called. It does not provide
exactly-once commit semantics.
+- Do not print `access_key_secret` in logs or job descriptions.
-> This example write data to the sls's logstore1.And if you have not yet
installed and deployed SeaTunnel, you need to follow the instructions in
Install SeaTunnel to install and deploy SeaTunnel. And if you have not yet
installed and deployed SeaTunnel, you need to follow the instructions in
[Install SeaTunnel](../../getting-started/locally/deployment.md) to install and
deploy SeaTunnel. And then follow the instructions in [Quick Start With
SeaTunnel Engine](../../getting-started/locall [...]
+## Task Example
-[Create RAM user and
authorization](https://help.aliyun.com/zh/sls/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service?spm=a2c4g.11186623.0.i4),Please
ensure thr ram user have sufficient rights to perform, reference [RAM Custom
Authorization
Example](https://help.aliyun.com/zh/sls/use-custom-policies-to-grant-permissions-to-a-ram-user?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#reference-s3z-m1l-z2b)
+### Write Rows to SLS
```hocon
-# Defining the runtime environment
env {
- parallelism = 2
- job.mode = "STREAMING"
- checkpoint.interval = 30000
+ parallelism = 1
+ job.mode = "BATCH"
}
+
source {
FakeSource {
row.num = 10
@@ -65,10 +68,10 @@ source {
string.length = 10
schema = {
fields = {
- id = "int"
- name = "string"
- description = "string"
- weight = "string"
+ id = "int"
+ name = "string"
+ description = "string"
+ weight = "string"
}
}
}
@@ -81,10 +84,13 @@ sink {
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ source = "seatunnel-demo"
+ topic = "fake-source"
+ log_group_size = 100
}
}
```
## Changelog
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/en/connectors/source/Sls.md b/docs/en/connectors/source/Sls.md
index 76b6231a50..161b00ca6a 100644
--- a/docs/en/connectors/source/Sls.md
+++ b/docs/en/connectors/source/Sls.md
@@ -8,7 +8,7 @@ import ChangeLog from '../changelog/connector-sls.md';
> Spark<br/>
> Flink<br/>
-> Seatunnel Zeta<br/>
+> SeaTunnel Zeta<br/>
## Key Features
@@ -21,12 +21,19 @@ import ChangeLog from '../changelog/connector-sls.md';
## Description
-Source connector for Aliyun Sls.
+The Sls source connector reads logs from Alibaba Cloud Simple Log Service
(SLS). It can run in batch or
+streaming jobs and reads SLS shards in parallel. In streaming mode, SeaTunnel
stores the SLS cursor
+during checkpoint completion, so a restarted job can continue from the
committed cursor.
+
+You can read logs in two ways:
+
+- Configure `schema` to parse named SLS log fields into SeaTunnel columns.
+- Omit `schema` to read each SLS log as one JSON string in a single `content`
column.
## Supported DataSource Info
-In order to use the Sls connector, the following dependencies are required.
-They can be downloaded via install-plugin.sh or from the Maven central
repository.
+To use the Sls connector, download the following dependency by using
`install-plugin.sh` or from the
+Maven central repository.
| Datasource | Supported Versions | Maven
|
|------------|--------------------|-----------------------------------------------------------------------------------|
@@ -34,31 +41,35 @@ They can be downloaded via install-plugin.sh or from the
Maven central repositor
## Source Options
-| Name | Type
| Required | Default |
Description
|
-|-------------------------------------|---------------------------------------------|----------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
-| project | String
| Yes | - | [Aliyun Sls
Project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project?spm=a2c4g.11186623.0.0.6f9755ebyfaYSl)
|
-| logstore | String
| Yes | - | [Aliyun Sls
Logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore?spm=a2c4g.11186623.0.0.13137c08nfuiBC)
|
-| endpoint | String
| Yes | - | [Aliyun Access
Endpoint](https://help.aliyun.com/zh/sls/developer-reference/api-sls-2020-12-30-endpoint?spm=a2c4g.11186623.0.0.548945a8UyJULa)
|
-| access_key_id | String
| Yes | - | [Aliyun AccessKey
ID](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| access_key_secret | String
| Yes | - | [Aliyun AccessKey
Secret](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| start_mode |
StartMode[earliest],[group_cursor],[latest] | No | group_cursor
| The initial consumption pattern of consumers.
|
-| consumer_group | String
| No | SeaTunnel-Consumer-Group | Sls consumer group id, used to
distinguish different consumer groups.
|
-| auto_cursor_reset | CursorMode[begin],[end]
| No | end | When there is no cursor in the
consumer group, cursor initialization occurs
|
-| batch_size | Int
| No | 1000 | The amount of data pulled from SLS
each time
|
-| partition-discovery.interval-millis | Long
| No | -1 | The interval for dynamically
discovering topics and partitions.
|
+| Name | Type
| Required | Default | Description
|
+|-------------------------------------|-------------------------------------------|----------|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------|
+| endpoint | String
| Yes | - | Alibaba Cloud SLS endpoint, for
example `cn-hangzhou.log.aliyuncs.com` or an intranet endpoint.
|
+| project | String
| Yes | - | [Alibaba Cloud SLS
project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project).
|
+| logstore | String
| Yes | - | [Alibaba Cloud SLS
logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore).
|
+| access_key_id | String
| Yes | - | Alibaba Cloud AccessKey ID.
|
+| access_key_secret | String
| Yes | - | Alibaba Cloud AccessKey secret.
|
+| start_mode | `earliest`, `group_cursor`, `latest`
| No | `group_cursor` | Initial cursor mode. `earliest`
starts from the beginning, `latest` starts from the end, and `group_cursor`
uses the consumer group's checkpoint. |
+| consumer_group | String
| No | `SeaTunnel-Consumer-Group` | SLS consumer group name. Use
different values when separate jobs must keep independent cursors.
|
+| auto_cursor_reset | `begin`, `end`
| No | `end` | Cursor position used when
`start_mode = group_cursor` but the consumer group has no checkpoint yet.
|
+| batch_size | Int
| No | 1000 | Maximum logs pulled from each shard
in one request.
|
+| partition-discovery.interval-millis | Long
| No | -1 | Interval for discovering SLS shard
changes. A value less than or equal to 0 disables periodic discovery.
|
+| schema | Config
| No | - | SeaTunnel schema for parsing SLS log
fields. If omitted, the connector outputs one `content` string column
containing the full log as JSON. |
-## Task Example
+## Notes
-### Simple
+- The configured RAM user must have permission to read the target project,
logstore, shards, consumer
+ groups, checkpoints, and logs.
+- In batch mode, the connector reads the currently assigned shards and then
finishes. Use streaming mode
+ for continuous log consumption.
+- Do not print `access_key_secret` in logs or job descriptions.
-> This example reads the data of sls's logstore1 and prints it to the
client.And if you have not yet installed and deployed SeaTunnel, you need to
follow the instructions in Install SeaTunnel to install and deploy SeaTunnel.
And if you have not yet installed and deployed SeaTunnel, you need to follow
the instructions in [Install
SeaTunnel](../../getting-started/locally/deployment.md) to install and deploy
SeaTunnel. And then follow the instructions in [Quick Start With SeaTunnel
Engine]( [...]
+## Task Examples
-[Create RAM user and
authorization](https://help.aliyun.com/zh/sls/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service?spm=a2c4g.11186623.0.i4),Please
ensure thr ram user have sufficient rights to perform, reference [RAM Custom
Authorization
Example](https://help.aliyun.com/zh/sls/use-custom-policies-to-grant-permissions-to-a-ram-user?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#reference-s3z-m1l-z2b)
+### Read Logs with an Explicit Schema
```hocon
-# Defining the runtime environment
env {
- parallelism = 2
+ parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 30000
}
@@ -70,21 +81,50 @@ source {
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ consumer_group = "seatunnel-sls-demo"
+ start_mode = "group_cursor"
+ auto_cursor_reset = "begin"
+ batch_size = 1000
schema = {
fields = {
- id = "int"
- name = "string"
- description = "string"
- weight = "string"
+ id = "int"
+ name = "string"
+ description = "string"
+ weight = "string"
}
}
}
}
sink {
- Console {
+ Console {}
+}
+```
+
+### Read Logs Without a Schema
+
+When `schema` is not configured, each output row has one `content` field. The
field value is a JSON
+string built from the SLS log contents.
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ Sls {
+ endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
+ project = "project1"
+ logstore = "logstore1"
+ access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
+ access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
+
+sink {
+ Console {}
+}
```
## Changelog
diff --git a/docs/zh/connectors/sink/Sls.md b/docs/zh/connectors/sink/Sls.md
index c6e7d4d2ee..3cb3845729 100644
--- a/docs/zh/connectors/sink/Sls.md
+++ b/docs/zh/connectors/sink/Sls.md
@@ -2,60 +2,62 @@ import ChangeLog from '../changelog/connector-sls.md';
# Sls
-> Sls sink connector
+> Sls Sink 连接器
-## Support Those Engines
+## 支持的引擎
> Spark<br/>
> Flink<br/>
-> Seatunnel Zeta<br/>
+> SeaTunnel Zeta<br/>
## 主要特性
-- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
-- [ ] [cdc](../../introduction/concepts/connector-v2-features.md)
-- [ ] [timer flush](../../introduction/concepts/connector-v2-features.md)
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [ ] [CDC](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 描述
-Sink connector for Aliyun Sls.
+Sls sink 连接器用于把 SeaTunnel 数据写入阿里云日志服务 SLS。每条 SeaTunnel 数据会先序列化为
+JSON，然后作为 SLS 日志项写入，日志内容的 key 为 `content`。
-从写入数据到阿里云Sls日志服务
+## 支持的数据源信息
-为了使用Sls连接器，需要以下依赖关系。
-它们可以通过install-plugin.sh或Maven中央存储库下载。
+使用 Sls 连接器前，需要通过 `install-plugin.sh` 或 Maven 中央仓库获取以下依赖。
-| Datasource | Supported Versions | Maven
|
-|------------|--------------------|-----------------------------------------------------------------------------------|
-| Sls | Universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls)
|
+| 数据源 | 支持版本 | Maven
|
+|--------|---------------|-----------------------------------------------------------------------------------|
+| Sls | Universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls)
|
-## 支持的数据源信息
+## Sink 选项
-| Name | Type | Required | Default
| Description
|
-|-------------------------------------|----------|----------|-------------------|------------------------------------------------------------------------------------------------------------------------------------|
-| project | String | Yes | -
| [阿里云 Sls
项目](https://help.aliyun.com/zh/sls/user-guide/manage-a-project?spm=a2c4g.11186623.0.0.6f9755ebyfaYSl)
|
-| logstore | String | Yes | -
| [阿里云 Sls
日志库](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore?spm=a2c4g.11186623.0.0.13137c08nfuiBC)
|
-| endpoint | String | Yes | -
|
[阿里云访问服务点](https://help.aliyun.com/zh/sls/developer-reference/api-sls-2020-12-30-endpoint?spm=a2c4g.11186623.0.0.548945a8UyJULa)
|
-| access_key_id | String | Yes | -
|
[阿里云访问用户ID](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| access_key_secret | String | Yes | -
|
[阿里云访问用户密码](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| source | String | No | SeaTunnel-Source
| 在sls中数据来源标记
|
-| topic | String | No | SeaTunnel-Topic
| 在sls中数据主题标记
|
+| 名称 | 类型 | 是否必填 | 默认值 | 描述
|
+|-------------------|--------|----------|--------------------|--------------------------------------------------------------------------------------------------------|
+| endpoint | String | 是 | - | 阿里云 SLS 访问地址，例如
`cn-hangzhou.log.aliyuncs.com` 或内网访问地址。 |
+| project | String | 是 | - | [阿里云 SLS
Project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project)。
|
+| logstore | String | 是 | - | [阿里云 SLS
Logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore)。
|
+| access_key_id | String | 是 | - | 阿里云 AccessKey
ID。
|
+| access_key_secret | String | 是 | - | 阿里云 AccessKey
Secret。
|
+| source | String | 否 | `SeaTunnel-Source` | 写入 SLS log group
的 source 标记。
|
+| topic | String | 否 | `SeaTunnel-Topic` | 写入 SLS log group
的 topic 标记。
|
+| log_group_size | Int | 否 | 100 | 配置的 SLS log
group 写入大小。当前 writer 每次写入请求会发送一条 SeaTunnel 数据。 |
-## 任务示例
+## 注意事项
-### 简单示例
+- 配置的 RAM 用户需要有向目标 project 和 logstore 写入日志的权限。
+- sink 在收到数据时立即写入，不提供精确一次提交语义。
+- 不要在日志或任务说明里打印 `access_key_secret`。
->
此示例写入sls的logstore1的数据。如果您尚未安装和部署SeaTunnel，则需要按照安装SeaTunnel中的说明安装和部署SeaTunnel。然后按照[快速启动SeaTunnel引擎](../../getting-started/locally/quick-start-seatunnel-engine.md)中的说明运行此作业。
+## 任务示例
-[创建RAM用户及授权](https://help.aliyun.com/zh/sls/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service?spm=a2c4g.11186623.0.i4),
请确认RAM用户有足够的权限来读取及管理数据，参考：[RAM自定义授权示例](https://help.aliyun.com/zh/sls/use-custom-policies-to-grant-permissions-to-a-ram-user?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#reference-s3z-m1l-z2b)
+### 写入数据到 SLS
```hocon
-# Defining the runtime environment
env {
- parallelism = 2
- job.mode = "STREAMING"
- checkpoint.interval = 30000
+ parallelism = 1
+ job.mode = "BATCH"
}
+
source {
FakeSource {
row.num = 10
@@ -65,10 +67,10 @@ source {
string.length = 10
schema = {
fields = {
- id = "int"
- name = "string"
- description = "string"
- weight = "string"
+ id = "int"
+ name = "string"
+ description = "string"
+ weight = "string"
}
}
}
@@ -81,6 +83,9 @@ sink {
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ source = "seatunnel-demo"
+ topic = "fake-source"
+ log_group_size = 100
}
}
```
diff --git a/docs/zh/connectors/source/Sls.md b/docs/zh/connectors/source/Sls.md
index a53b5b693c..54db0f1538 100644
--- a/docs/zh/connectors/source/Sls.md
+++ b/docs/zh/connectors/source/Sls.md
@@ -2,13 +2,13 @@ import ChangeLog from '../changelog/connector-sls.md';
# Sls
-> Sls source connector
+> Sls 源连接器
## 支持的引擎
> Spark<br/>
> Flink<br/>
-> Seatunnel Zeta<br/>
+> SeaTunnel Zeta<br/>
## 主要特性
@@ -17,48 +17,56 @@ import ChangeLog from '../changelog/connector-sls.md';
- [x] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [ ] [列投影](../../introduction/concepts/connector-v2-features.md)
- [x] [并行度](../../introduction/concepts/connector-v2-features.md)
-- [ ] [支持用户定义的拆分](../../introduction/concepts/connector-v2-features.md)
+- [ ] [支持用户自定义切分](../../introduction/concepts/connector-v2-features.md)
## 描述
-从阿里云Sls日志服务中读取数据。
+Sls source 连接器用于从阿里云日志服务 SLS 读取日志。它支持批处理和流处理任务，并且可以按
+SLS shard 并行读取。流处理模式下，SeaTunnel 会在 checkpoint 完成时提交 SLS 游标，任务重启后
+可以从已经提交的位置继续读取。
+
+可以用两种方式读取日志：
+
+- 配置 `schema`，把指定的 SLS 日志字段解析成 SeaTunnel 字段。
+- 不配置 `schema`，把每条 SLS 日志作为 JSON 字符串写入单个 `content` 字段。
## 支持的数据源信息
-为了使用Sls连接器，需要以下依赖关系。
-它们可以通过install-plugin.sh或Maven中央存储库下载。
+使用 Sls 连接器前，需要通过 `install-plugin.sh` 或 Maven 中央仓库获取以下依赖。
-| 数据源 | 支持的版本 | Maven
|
-|-----|-----------|-----------------------------------------------------------------------------------|
-| Sls | Universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls)
|
+| 数据源 | 支持版本 | Maven
|
+|--------|---------------|-----------------------------------------------------------------------------------|
+| Sls | Universal |
[Download](https://mvnrepository.com/artifact/org.apache.seatunnel/connector-sls)
|
-## Source Options
+## Source 选项
-| Name | Type
| Required | Default |
Description
|
-|-------------------------------------|---------------------------------------------|----------|--------------------------|------------------------------------------------------------------------------------------------------------------------------------|
-| project | String
| Yes | - | [阿里云 Sls
项目](https://help.aliyun.com/zh/sls/user-guide/manage-a-project?spm=a2c4g.11186623.0.0.6f9755ebyfaYSl)
|
-| logstore | String
| Yes | - | [阿里云 Sls
日志库](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore?spm=a2c4g.11186623.0.0.13137c08nfuiBC)
|
-| endpoint | String
| Yes | - |
[阿里云访问服务点](https://help.aliyun.com/zh/sls/developer-reference/api-sls-2020-12-30-endpoint?spm=a2c4g.11186623.0.0.548945a8UyJULa)
|
-| access_key_id | String
| Yes | - |
[阿里云访问用户ID](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| access_key_secret | String
| Yes | - |
[阿里云访问用户密码](https://help.aliyun.com/zh/ram/user-guide/create-an-accesskey-pair?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#task-2245479)
|
-| start_mode |
StartMode[earliest],[group_cursor],[latest] | No | group_cursor
| 消费者的初始消费模式
|
-| consumer_group | String
| No | SeaTunnel-Consumer-Group | Sls消费者组id，用于区分不同的消费者组
|
-| auto_cursor_reset | CursorMode[begin],[end]
| No | end | 当消费者组中没有记录读取游标时，初始化读取游标
|
-| batch_size | Int
| No | 1000 | 每次从SLS中读取的数据量
|
-| partition-discovery.interval-millis | Long
| No | -1 | 动态发现主题和分区的间隔
|
+| 名称 | 类型
| 是否必填 | 默认值 | 描述
|
+|-------------------------------------|-------------------------------------------|----------|----------------------------|----------------------------------------------------------------------------------------------------------------|
+| endpoint | String
| 是 | - | 阿里云 SLS 访问地址，例如
`cn-hangzhou.log.aliyuncs.com` 或内网访问地址。 |
+| project | String
| 是 | - | [阿里云 SLS
Project](https://help.aliyun.com/zh/sls/user-guide/manage-a-project)。
|
+| logstore | String
| 是 | - | [阿里云 SLS
Logstore](https://help.aliyun.com/zh/sls/user-guide/manage-a-logstore)。
|
+| access_key_id | String
| 是 | - | 阿里云 AccessKey ID。
|
+| access_key_secret | String
| 是 | - | 阿里云 AccessKey Secret。
|
+| start_mode | `earliest`, `group_cursor`, `latest`
| 否 | `group_cursor` | 初始读取位置。`earliest` 从最早位置读取，`latest`
从最新位置读取，`group_cursor` 使用消费者组已提交的游标。 |
+| consumer_group | String
| 否 | `SeaTunnel-Consumer-Group` | SLS
消费者组名称。不同任务如果需要独立保存读取位置，应该使用不同的消费者组。 |
+| auto_cursor_reset | `begin`, `end`
| 否 | `end` | 当 `start_mode = group_cursor`
但消费者组还没有已提交游标时，使用的初始化位置。 |
+| batch_size | Int
| 否 | 1000 | 每次从每个 shard 拉取的最大日志条数。
|
+| partition-discovery.interval-millis | Long
| 否 | -1 | 动态发现 SLS shard 变化的间隔。小于或等于 0
表示不定时发现。 |
+| schema | Config
| 否 | - | 用于解析 SLS 日志字段的 SeaTunnel
表结构。不配置时输出单个 `content` 字符串字段，字段值是整条日志的 JSON 内容。 |
-## 任务示例
+## 注意事项
-### 简单示例
+- 配置的 RAM 用户需要有读取目标 project、logstore、shard、消费者组、checkpoint 和日志数据的权限。
+- 批处理模式会读取当前分配的 shard 后结束；持续消费日志请使用流处理模式。
+- 不要在日志或任务说明里打印 `access_key_secret`。
->
此示例读取sls的logstore1的数据并将其打印到客户端。如果您尚未安装和部署SeaTunnel，则需要按照安装SeaTunnel中的说明安装和部署SeaTunnel。然后按照[快速启动SeaTunnel引擎](../../getting-started/locally/quick-start-seatunnel-engine.md)中的说明运行此作业。
+## 任务示例
-[创建RAM用户及授权](https://help.aliyun.com/zh/sls/create-a-ram-user-and-authorize-the-ram-user-to-access-log-service?spm=a2c4g.11186623.0.i4),
请确认RAM用户有足够的权限来读取及管理数据，参考：[RAM自定义授权示例](https://help.aliyun.com/zh/sls/use-custom-policies-to-grant-permissions-to-a-ram-user?spm=a2c4g.11186623.0.0.4a6e4e554CKhSc#reference-s3z-m1l-z2b)
+### 配置 Schema 读取日志
```hocon
-# Defining the runtime environment
env {
- parallelism = 2
+ parallelism = 1
job.mode = "STREAMING"
checkpoint.interval = 30000
}
@@ -70,21 +78,49 @@ source {
logstore = "logstore1"
access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
+ consumer_group = "seatunnel-sls-demo"
+ start_mode = "group_cursor"
+ auto_cursor_reset = "begin"
+ batch_size = 1000
schema = {
fields = {
- id = "int"
- name = "string"
- description = "string"
- weight = "string"
+ id = "int"
+ name = "string"
+ description = "string"
+ weight = "string"
}
}
}
}
sink {
- Console {
+ Console {}
+}
+```
+
+### 不配置 Schema 读取日志
+
+不配置 `schema` 时，输出结果只有一个 `content` 字段，字段值是由 SLS 日志内容组成的 JSON 字符串。
+
+```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ Sls {
+ endpoint = "cn-hangzhou-intranet.log.aliyuncs.com"
+ project = "project1"
+ logstore = "logstore1"
+ access_key_id = "xxxxxxxxxxxxxxxxxxxxxxxx"
+ access_key_secret = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
}
}
+
+sink {
+ Console {}
+}
```
## 变更日志
