This is an automated email from the ASF dual-hosted git repository.
davidzollo pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new e870a7c389 [Docs][Connector-V2] Improve Airtable Github and Gitlab
docs (#11345)
e870a7c389 is described below
commit e870a7c389a1a2d59ba9b226cd0f763bba2037ea
Author: Daniel Carter <[email protected]>
AuthorDate: Sun Jul 5 11:19:22 2026 +0800
[Docs][Connector-V2] Improve Airtable Github and Gitlab docs (#11345)
Co-authored-by: DanielCarter-stack
<[email protected]>
---
docs/en/connectors/sink/Airtable.md | 29 ++++
docs/en/connectors/source/Airtable.md | 48 ++++++
docs/en/connectors/source/Github.md | 284 +++++++++----------------------
docs/en/connectors/source/Gitlab.md | 278 ++++++++----------------------
docs/zh/connectors/sink/Airtable.md | 32 +++-
docs/zh/connectors/source/Airtable.md | 54 +++++-
docs/zh/connectors/source/Github.md | 307 ++++++++++------------------------
docs/zh/connectors/source/Gitlab.md | 289 +++++++++-----------------------
8 files changed, 476 insertions(+), 845 deletions(-)
diff --git a/docs/en/connectors/sink/Airtable.md
b/docs/en/connectors/sink/Airtable.md
index 8d8cb1b959..09f4861a41 100644
--- a/docs/en/connectors/sink/Airtable.md
+++ b/docs/en/connectors/sink/Airtable.md
@@ -72,7 +72,35 @@ Sink plugin common parameters, please refer to [Sink Common
Options](../common-o
## Example
+Write rows to an Airtable table:
+
```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ FakeSource {
+ schema = {
+ fields {
+ Name = string
+ Age = int
+ }
+ }
+ rows = [
+ {
+ kind = INSERT
+ fields = ["Alice", 30]
+ },
+ {
+ kind = INSERT
+ fields = ["Bob", 25]
+ }
+ ]
+ }
+}
+
sink {
Airtable {
token = "patXXXXXXXX.XXXXXXXX"
@@ -80,6 +108,7 @@ sink {
table = "Shipments"
typecast = true
batch_size = 10
+ request_interval_ms = 220
}
}
```
diff --git a/docs/en/connectors/source/Airtable.md
b/docs/en/connectors/source/Airtable.md
index 61acae4983..5e09b47b89 100644
--- a/docs/en/connectors/source/Airtable.md
+++ b/docs/en/connectors/source/Airtable.md
@@ -36,6 +36,10 @@ Used to read data from Airtable.
| record_metadata | List | No | - |
| time_zone | String | No | - |
| user_locale | String | No | - |
+| offset | String | No | - |
+| headers | Map | No | - |
+| body | String | No | - |
+| pageing | Config | No | - |
| request_interval_ms | int | No | 220 |
| rate_limit_backoff_ms | int | No | 30000 |
| rate_limit_max_retries | int | No | 3 |
@@ -44,6 +48,7 @@ Used to read data from Airtable.
| format | String | No | text |
| content_field | String | No | - |
| json_field | Config | No | - |
+| json_filed_missed_return_null | boolean | No | false |
| common-options | config | No | - |
### token [String]
@@ -106,6 +111,22 @@ The time zone for formatting date/time values.
The user locale for formatting values.
+### offset [String]
+
+Pagination offset returned by Airtable. Usually you do not need to set this
manually because the connector follows Airtable pagination automatically.
+
+### headers [Map]
+
+Extra HTTP headers. The connector automatically adds Airtable authorization
and JSON content type headers.
+
+### body [String]
+
+Advanced request body. Do not use it together with dedicated Airtable request
options such as `fields`, `filter_by_formula`, `page_size`, or `sort` for the
same Airtable API key.
+
+### pageing [Config]
+
+HTTP pagination configuration inherited from the HTTP connector. For normal
Airtable list-records reads, prefer Airtable's own pagination handled by the
connector.
+
### request_interval_ms [int]
Minimum interval in milliseconds between API requests. Default 220ms (to stay
within Airtable's 5 requests/second limit).
@@ -136,6 +157,10 @@ JsonPath expression to extract data from the response. For
Airtable, you typical
This parameter helps you configure the schema and must be used with schema.
+### json_filed_missed_return_null [boolean]
+
+When `true`, missing JSON fields return `null`; otherwise a missing field
causes an error.
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](../common-options/source-common-options.md) for details.
@@ -177,6 +202,29 @@ source {
}
```
+Read a small page from Airtable:
+
+```hocon
+source {
+ Airtable {
+ token = "patXXXXXXXX.XXXXXXXX"
+ base_id = "appXXXXXXXX"
+ table = "Shipments"
+ format = "json"
+ content_field = "$.records[*].fields"
+ page_size = 2
+ request_interval_ms = 220
+ schema = {
+ fields {
+ Name = string
+ Age = int
+ Status = string
+ }
+ }
+ }
+}
+```
+
## Changelog
<ChangeLog />
diff --git a/docs/en/connectors/source/Github.md
b/docs/en/connectors/source/Github.md
index 59131b3b11..1ed1d9392f 100644
--- a/docs/en/connectors/source/Github.md
+++ b/docs/en/connectors/source/Github.md
@@ -6,12 +6,12 @@ import ChangeLog from '../changelog/connector-http-github.md';
## Description
-Used to read data from Github.
+The Github source connector reads data from the GitHub REST API. It is built
on the HTTP source connector, and automatically adds the `Authorization: Bearer
<access_token>` request header.
## Key features
- [x] [batch](../../introduction/concepts/connector-v2-features.md)
-- [ ] [stream](../../introduction/concepts/connector-v2-features.md)
+- [x] [stream](../../introduction/concepts/connector-v2-features.md)
- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
- [ ] [column projection](../../introduction/concepts/connector-v2-features.md)
- [ ] [parallelism](../../introduction/concepts/connector-v2-features.md)
@@ -19,276 +19,145 @@ Used to read data from Github.
## Options
-| name | type | required | default value |
+| name | type | required | default value |
|-----------------------------|---------|----------|---------------|
| url | String | Yes | - |
-| access_token | String | No | - |
-| method | String | No | get |
-| schema.fields | Config | No | - |
-| format | String | No | json |
+| access_token | String | Yes | - |
+| method | String | No | GET |
+| headers | Map | No | - |
| params | Map | No | - |
| body | String | No | - |
+| format | String | No | text |
+| schema | Config | No | - |
+| schema.fields | Config | No | - |
| json_field | Config | No | - |
-| content_json | String | No | - |
+| content_field | String | No | - |
+| pageing | Config | No | - |
| poll_interval_millis | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
| retry_backoff_max_ms | int | No | 10000 |
-| enable_multi_lines | boolean | No | false |
+| json_filed_missed_return_null | boolean | No | false |
| common-options | config | No | - |
### url [String]
-http request url
+GitHub REST API URL, for example `https://api.github.com/orgs/apache/repos`.
### access_token [String]
-Github personal access token, see: [Creating a personal access token - GitHub
Docs](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
+GitHub personal access token. The connector sends it as a Bearer token in the
HTTP `Authorization` header.
### method [String]
-http request method, only supports GET, POST method
-
-### params [Map]
-
-http params
-
-### body [String]
-
-http body
-
-### poll_interval_millis [int]
-
-request http api interval(millis) in stream mode
+HTTP request method. The common GitHub read scenario uses `GET`.
-### retry [int]
+### headers [Map]
-The max retry times if request http return to `IOException`
+Extra HTTP headers. Do not put `Authorization` here unless you intentionally
want to override the header generated from `access_token`.
-### retry_backoff_multiplier_ms [int]
+### params [Map]
-The retry-backoff times(millis) multiplier if request http failed
+HTTP query parameters, such as `per_page`, `page`, `since`, or other GitHub
API parameters.
-### retry_backoff_max_ms [int]
+### body [String]
-The maximum retry-backoff times(millis) if request http failed
+HTTP request body. This is only useful for API endpoints that accept a request
body.
### format [String]
-the format of upstream data, now only support `json` `text`, default `json`.
-
-when you assign format is `json`, you should also assign schema option, for
example:
-
-upstream data is the following:
+Response format. Supports `json` and `text`. Use `json` with `schema` when you
want SeaTunnel rows with named fields.
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
-
-you should assign schema as the following:
-
-```hocon
-
-schema {
- fields {
- code = int
- data = string
- success = boolean
- }
-}
-
-```
+### schema [Config]
-connector will generate data as the following:
+Defines the output row structure when `format = "json"`. For details, see
[Schema Feature](../../introduction/concepts/schema-feature.md).
-| code | data | success |
-|------|-------------|---------|
-| 200 | get success | true |
+### json_field [Config]
-when you assign format is `text`, connector will do nothing for upstream data,
for example:
+Maps output fields to JSONPath expressions. Use it with `schema` when the
required values are nested in the response.
-upstream data is the following:
+### content_field [String]
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
+JSONPath expression used to select a JSON fragment before schema parsing, for
example `$.items[*]`.
-connector will generate data as the following:
+### pageing [Config]
-| content |
-|----------------------------------------------------------|
-| {"code": 200, "data": "get success", "success": true} |
+Pagination settings inherited from the HTTP connector. Keep the option name
`pageing` in job configs.
-### schema [Config]
-
-#### fields [Config]
+### poll_interval_millis [int]
-the schema fields of upstream data
+Request interval in milliseconds for streaming jobs. In batch jobs the
connector reads once and finishes.
-### content_json [String]
+### retry [int]
-This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+Maximum retry count when an HTTP request fails with `IOException`.
-If your return data looks something like this.
+### retry_backoff_multiplier_ms [int]
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+Retry backoff multiplier in milliseconds.
-You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
-
-```json
-[
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
-]
-```
+### retry_backoff_max_ms [int]
-Then you can get the desired result with a simpler schema,like
+Maximum retry backoff in milliseconds.
-```hocon
-Http {
- url = "http://mockserver:1080/contentjson/mock";
- method = "GET"
- format = "json"
- content_field = "$.store.book.*"
- schema = {
- fields {
- category = string
- author = string
- title = string
- price = string
- }
- }
-}
-```
+### json_filed_missed_return_null [boolean]
-Here is an example:
+When `true`, missing JSON fields return `null`; otherwise a missing field
causes an error.
-- Test data can be found at this link
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
-- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+### common options
-### json_field [Config]
+Source plugin common parameters. See [Source Common
Options](../common-options/source-common-options.md).
-This parameter helps you configure the schema,so this parameter must be used
with schema.
-
-If your data looks something like this:
-
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+## Example
-You can get the contents of 'book' by configuring the task as follows:
+Read repositories from a GitHub organization:
```hocon
source {
- Http {
- url = "http://mockserver:1080/jsonpath/mock";
+ Github {
+ url = "https://api.github.com/orgs/apache/repos";
+ access_token = "ghp_xxxxxxxxxxxx"
method = "GET"
format = "json"
- json_field = {
- category = "$.store.book[*].category"
- author = "$.store.book[*].author"
- title = "$.store.book[*].title"
- price = "$.store.book[*].price"
- }
schema = {
fields {
- category = string
- author = string
- title = string
- price = string
+ id = int
+ name = string
+ description = string
+ html_url = string
+ stargazers_count = int
+ forks = int
}
}
}
}
```
-- Test data can be found at this link
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
-- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
-
-### common options
-
-Source plugin common parameters, please refer to [Source Common
Options](../common-options/source-common-options.md) for details
-
-## Example
+Read paged GitHub API results:
```hocon
-Github {
- url = "https://api.github.com/orgs/apache/repos";
- access_token = "xxxx"
- method = "GET"
- format = "json"
- schema = {
- fields {
- id = int
- name = string
- description = string
- html_url = string
- stargazers_count = int
- forks = int
+source {
+ Github {
+ url = "https://api.github.com/orgs/apache/repos";
+ access_token = "ghp_xxxxxxxxxxxx"
+ method = "GET"
+ params = {
+ per_page = "100"
+ page = "${page}"
+ }
+ pageing = {
+ page_field = "page"
+ total_page_size = 5
+ start_page_number = 1
+ use_placeholder_replacement = true
+ }
+ format = "json"
+ schema = {
+ fields {
+ id = int
+ name = string
+ html_url = string
+ }
}
}
}
@@ -297,4 +166,3 @@ Github {
## Changelog
<ChangeLog />
-
diff --git a/docs/en/connectors/source/Gitlab.md
b/docs/en/connectors/source/Gitlab.md
index 024b2a3256..1bdcec8c03 100644
--- a/docs/en/connectors/source/Gitlab.md
+++ b/docs/en/connectors/source/Gitlab.md
@@ -6,7 +6,7 @@ import ChangeLog from '../changelog/connector-http-gitlab.md';
## Description
-Used to read data from Gitlab.
+The Gitlab source connector reads data from the GitLab REST API. It is built
on the HTTP source connector, and automatically sends `access_token` as the
GitLab `PRIVATE-TOKEN` request header.
## Key features
@@ -19,280 +19,150 @@ Used to read data from Gitlab.
## Options
-| name | type | required | default value |
+| name | type | required | default value |
|-----------------------------|---------|----------|---------------|
| url | String | Yes | - |
| access_token | String | Yes | - |
-| method | String | No | get |
-| schema.fields | Config | No | - |
-| format | String | No | json |
+| method | String | No | GET |
+| headers | Map | No | - |
| params | Map | No | - |
| body | String | No | - |
+| format | String | No | text |
+| schema | Config | No | - |
+| schema.fields | Config | No | - |
| json_field | Config | No | - |
-| content_json | String | No | - |
+| content_field | String | No | - |
+| pageing | Config | No | - |
| poll_interval_millis | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
| retry_backoff_max_ms | int | No | 10000 |
-| enable_multi_lines | boolean | No | false |
+| json_filed_missed_return_null | boolean | No | false |
| common-options | config | No | - |
### url [String]
-http request url
+GitLab REST API URL, for example `https://gitlab.com/api/v4/projects`.
### access_token [String]
-personal access token
+GitLab personal access token. The connector sends it in the HTTP
`PRIVATE-TOKEN` header.
### method [String]
-http request method, only supports GET, POST method
-
-### params [Map]
-
-http params
+HTTP request method. The common GitLab read scenario uses `GET`.
-### body [String]
-
-http body
+### headers [Map]
-### poll_interval_millis [int]
+Extra HTTP headers. Do not put `PRIVATE-TOKEN` here unless you intentionally
want to override the header generated from `access_token`.
-request http api interval(millis) in stream mode
-
-### retry [int]
-
-The max retry times if request http return to `IOException`
-
-### retry_backoff_multiplier_ms [int]
+### params [Map]
-The retry-backoff times(millis) multiplier if request http failed
+HTTP query parameters, such as `per_page`, `page`, `owned`, or other GitLab
API parameters.
-### retry_backoff_max_ms [int]
+### body [String]
-The maximum retry-backoff times(millis) if request http failed
+HTTP request body. This is only useful for API endpoints that accept a request
body.
### format [String]
-the format of upstream data, now only support `json` `text`, default `json`.
-
-when you assign format is `json`, you should also assign schema option, for
example:
+Response format. Supports `json` and `text`. Use `json` with `schema` when you
want SeaTunnel rows with named fields.
-upstream data is the following:
-
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
-
-you should assign schema as the following:
-
-```hocon
-
-schema {
- fields {
- code = int
- data = string
- success = boolean
- }
-}
-
-```
+### schema [Config]
-connector will generate data as the following:
+Defines the output row structure when `format = "json"`. For details, see
[Schema Feature](../../introduction/concepts/schema-feature.md).
-| code | data | success |
-|------|-------------|---------|
-| 200 | get success | true |
+### json_field [Config]
-when you assign format is `text`, connector will do nothing for upstream data,
for example:
+Maps output fields to JSONPath expressions. Use it with `schema` when the
required values are nested in the response.
-upstream data is the following:
+### content_field [String]
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
+JSONPath expression used to select a JSON fragment before schema parsing, for
example `$.items[*]`.
-connector will generate data as the following:
+### pageing [Config]
-| content |
-|----------------------------------------------------------|
-| {"code": 200, "data": "get success", "success": true} |
+Pagination settings inherited from the HTTP connector. Keep the option name
`pageing` in job configs.
-### schema [Config]
+### poll_interval_millis [int]
-#### fields [Config]
+This option is inherited from the HTTP connector, but Gitlab source currently
supports batch mode only.
-the schema fields of upstream data
+### retry [int]
-### content_json [String]
+Maximum retry count when an HTTP request fails with `IOException`.
-This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+### retry_backoff_multiplier_ms [int]
-If your return data looks something like this.
+Retry backoff multiplier in milliseconds.
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
-
-You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
-
-```json
-[
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
-]
-```
+### retry_backoff_max_ms [int]
-Then you can get the desired result with a simpler schema,like
+Maximum retry backoff in milliseconds.
-```hocon
-Http {
- url = "http://mockserver:1080/contentjson/mock";
- method = "GET"
- format = "json"
- content_field = "$.store.book.*"
- schema = {
- fields {
- category = string
- author = string
- title = string
- price = string
- }
- }
-}
-```
+### json_filed_missed_return_null [boolean]
-Here is an example:
+When `true`, missing JSON fields return `null`; otherwise a missing field
causes an error.
-- Test data can be found at this link
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
-- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+### common options
-### json_field [Config]
+Source plugin common parameters. See [Source Common
Options](../common-options/source-common-options.md).
-This parameter helps you configure the schema,so this parameter must be used
with schema.
-
-If your data looks something like this:
-
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+## Example
-You can get the contents of 'book' by configuring the task as follows:
+Read projects from GitLab:
```hocon
source {
- Http {
- url = "http://mockserver:1080/jsonpath/mock";
+ Gitlab {
+ url = "https://gitlab.com/api/v4/projects";
+ access_token = "glpat-xxxxxxxxxxxx"
method = "GET"
format = "json"
- json_field = {
- category = "$.store.book[*].category"
- author = "$.store.book[*].author"
- title = "$.store.book[*].title"
- price = "$.store.book[*].price"
- }
schema = {
fields {
- category = string
- author = string
- title = string
- price = string
+ id = int
+ description = string
+ name = string
+ name_with_namespace = string
+ path = string
+ http_url_to_repo = string
}
}
}
}
```
-- Test data can be found at this link
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
-- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
-
-### common options
-
-Source plugin common parameters, please refer to [Source Common
Options](../common-options/source-common-options.md) for details
-
-## Example
+Read paged GitLab API results:
```hocon
-Gitlab{
+source {
+ Gitlab {
url = "https://gitlab.com/api/v4/projects";
- access_token = "xxxxx"
- schema {
- fields {
- id = int
- description = string
- name = string
- name_with_namespace = string
- path = string
- http_url_to_repo = string
- }
+ access_token = "glpat-xxxxxxxxxxxx"
+ method = "GET"
+ params = {
+ per_page = "100"
+ page = "${page}"
+ }
+ pageing = {
+ page_field = "page"
+ total_page_size = 5
+ start_page_number = 1
+ use_placeholder_replacement = true
}
+ format = "json"
+ schema = {
+ fields {
+ id = int
+ name = string
+ path = string
+ }
+ }
+ }
}
```
## Changelog
<ChangeLog />
-
diff --git a/docs/zh/connectors/sink/Airtable.md
b/docs/zh/connectors/sink/Airtable.md
index cc8c1e49d1..2246ecad98 100644
--- a/docs/zh/connectors/sink/Airtable.md
+++ b/docs/zh/connectors/sink/Airtable.md
@@ -13,6 +13,7 @@ import ChangeLog from
'../changelog/connector-http-airtable.md';
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [ ] [cdc](../../introduction/concepts/connector-v2-features.md)
- [ ] [支持多表写入](../../introduction/concepts/connector-v2-features.md)
+- [ ] [定时刷新](../../introduction/concepts/connector-v2-features.md)
## 选项
@@ -67,11 +68,39 @@ API 请求之间的最小间隔（毫秒），默认 220ms。
### common options
-汇插件通用参数，请参考 [Sink Common Options](../common-options/sink-common-options.md)。
+Sink 插件通用参数，请参考 [Sink Common
Options](../common-options/sink-common-options.md)。
## 示例
+将数据写入 Airtable 表：
+
```hocon
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ FakeSource {
+ schema = {
+ fields {
+ Name = string
+ Age = int
+ }
+ }
+ rows = [
+ {
+ kind = INSERT
+ fields = ["Alice", 30]
+ },
+ {
+ kind = INSERT
+ fields = ["Bob", 25]
+ }
+ ]
+ }
+}
+
sink {
Airtable {
token = "patXXXXXXXX.XXXXXXXX"
@@ -79,6 +108,7 @@ sink {
table = "Shipments"
typecast = true
batch_size = 10
+ request_interval_ms = 220
}
}
```
diff --git a/docs/zh/connectors/source/Airtable.md
b/docs/zh/connectors/source/Airtable.md
index 19bbb70df1..9fd33229e0 100644
--- a/docs/zh/connectors/source/Airtable.md
+++ b/docs/zh/connectors/source/Airtable.md
@@ -8,14 +8,14 @@ import ChangeLog from
'../changelog/connector-http-airtable.md';
用于从 Airtable 读取数据。
-## 关键特性
+## 主要特性
- [x] [批](../../introduction/concepts/connector-v2-features.md)
- [ ] [流](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [x] [列投影](../../introduction/concepts/connector-v2-features.md)
-- [ ] [并行性](../../introduction/concepts/connector-v2-features.md)
-- [ ] [支持用户自定义split](../../introduction/concepts/connector-v2-features.md)
+- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
+- [ ] [支持用户自定义分片](../../introduction/concepts/connector-v2-features.md)
## 选项
@@ -36,6 +36,10 @@ import ChangeLog from
'../changelog/connector-http-airtable.md';
| record_metadata | List | 否 | - |
| time_zone | String | 否 | - |
| user_locale | String | 否 | - |
+| offset | String | 否 | - |
+| headers | Map | 否 | - |
+| body | String | 否 | - |
+| pageing | Config | 否 | - |
| request_interval_ms | int | 否 | 220 |
| rate_limit_backoff_ms | int | 否 | 30000 |
| rate_limit_max_retries | int | 否 | 3 |
@@ -44,6 +48,7 @@ import ChangeLog from
'../changelog/connector-http-airtable.md';
| format | String | 否 | text |
| content_field | String | 否 | - |
| json_field | Config | 否 | - |
+| json_filed_missed_return_null | boolean | 否 | false |
| common-options | config | 否 | - |
### token [String]
@@ -106,6 +111,22 @@ Airtable 公式表达式，用于过滤记录。参考 [Airtable 公式文档](h
用于格式化值的用户区域设置。
+### offset [String]
+
+Airtable 返回的分页偏移量。通常不需要手动配置，连接器会自动继续读取 Airtable 后续分页。
+
+### headers [Map]
+
+额外的 HTTP 请求头。连接器会自动添加 Airtable 认证头和 JSON 内容类型请求头。
+
+### body [String]
+
+高级请求体配置。不要把它和 `fields`、`filter_by_formula`、`page_size`、`sort` 等专用 Airtable
请求选项配置成同一个 Airtable API 字段。
+
+### pageing [Config]
+
+继承自 HTTP 连接器的分页配置。普通 Airtable 读取建议优先使用连接器内置的 Airtable 分页处理。
+
### request_interval_ms [int]
API 请求之间的最小间隔（毫秒），默认 220ms（以保持在 Airtable 每秒 5 次请求的限制内）。
@@ -136,6 +157,10 @@ API 请求之间的最小间隔（毫秒），默认 220ms（以保持在 Airtab
此参数帮助您配置模式，必须与 schema 一起使用。
+### json_filed_missed_return_null [boolean]
+
+设置为 `true` 时，JSON 字段缺失会返回 `null`；否则字段缺失会报错。
+
### common options
源插件通用参数，请参考 [Source Common
Options](../common-options/source-common-options.md)。
@@ -177,6 +202,29 @@ source {
}
```
+读取 Airtable 的小批量分页数据：
+
+```hocon
+source {
+ Airtable {
+ token = "patXXXXXXXX.XXXXXXXX"
+ base_id = "appXXXXXXXX"
+ table = "Shipments"
+ format = "json"
+ content_field = "$.records[*].fields"
+ page_size = 2
+ request_interval_ms = 220
+ schema = {
+ fields {
+ Name = string
+ Age = int
+ Status = string
+ }
+ }
+ }
+}
+```
+
## 变更日志
<ChangeLog />
diff --git a/docs/zh/connectors/source/Github.md
b/docs/zh/connectors/source/Github.md
index 2120f4ecea..84d4cafe67 100644
--- a/docs/zh/connectors/source/Github.md
+++ b/docs/zh/connectors/source/Github.md
@@ -6,12 +6,12 @@ import ChangeLog from '../changelog/connector-http-github.md';
## 描述
-用于从 Github 读取数据。
+Github 源连接器用于读取 GitHub REST API 数据。它基于 HTTP 源连接器实现，并会自动把 `access_token` 组装成
`Authorization: Bearer <access_token>` 请求头。
-## 关键特性
+## 主要特性
- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
-- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
+- [x] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [ ] [列投影](../../introduction/concepts/connector-v2-features.md)
- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
@@ -19,276 +19,145 @@ import ChangeLog from
'../changelog/connector-http-github.md';
## 选项
-| 名称 | 类型 | 必填 | 默认值 |
-|---------------------------|----------|------|--------|
-| url | String | 是 | - |
-| access_token | String | 否 | - |
-| method | String | 否 | get |
-| schema.fields | Config | 否 | - |
-| format | String | 否 | json |
-| params | Map | 否 | - |
-| body | String | 否 | - |
-| json_field | Config | 否 | - |
-| content_json | String | 否 | - |
-| poll_interval_millis | int | 否 | - |
-| retry | int | 否 | - |
-| retry_backoff_multiplier_ms | int | 否 | 100 |
-| retry_backoff_max_ms | int | 否 | 10000 |
-| enable_multi_lines | boolean | 否 | false |
-| common-options | config | 否 | - |
+| 参数名 | 类型 | 必填 | 默认值 |
+|-----------------------------|---------|------|--------|
+| url | String | 是 | - |
+| access_token | String | 是 | - |
+| method | String | 否 | GET |
+| headers | Map | 否 | - |
+| params | Map | 否 | - |
+| body | String | 否 | - |
+| format | String | 否 | text |
+| schema | Config | 否 | - |
+| schema.fields | Config | 否 | - |
+| json_field | Config | 否 | - |
+| content_field | String | 否 | - |
+| pageing | Config | 否 | - |
+| poll_interval_millis | int | 否 | - |
+| retry | int | 否 | - |
+| retry_backoff_multiplier_ms | int | 否 | 100 |
+| retry_backoff_max_ms | int | 否 | 10000 |
+| json_filed_missed_return_null | boolean | 否 | false |
+| common-options | config | 否 | - |
### url [String]
-HTTP 请求 URL。
+GitHub REST API 地址，例如 `https://api.github.com/orgs/apache/repos`。
### access_token [String]
-GitHub个人访问令牌，请参阅：[创建个人访问令牌 -
Github文档](https://docs.github.com/en/authentication/keeping-your-account-and-data-secure/creating-a-personal-access-token)
+GitHub 个人访问令牌。连接器会把它作为 Bearer token 写入 HTTP `Authorization` 请求头。
### method [String]
-HTTP 请求方法。目前支持 `GET` 和 `POST`。
+HTTP 请求方法。常见的 GitHub 读取场景使用 `GET`。
-### params [Map]
-
-http 参数
-
-### body [String]
-
-HTTP 请求体
+### headers [Map]
-### poll_interval_millis [int]
-
-流模式下请求 API 的间隔时间（毫秒）。
-
-### retry [int]
+额外的 HTTP 请求头。除非你确实想覆盖由 `access_token` 生成的认证头，否则不要在这里配置 `Authorization`。
-请求失败（`IOException`）时最大重试次数。
-
-### retry_backoff_multiplier_ms [int]
+### params [Map]
-请求失败时的退避时间（毫秒）乘数。
+HTTP 查询参数，例如 `per_page`、`page`、`since` 或其他 GitHub API 参数。
-### retry_backoff_max_ms [int]
+### body [String]
-请求失败时的最大退避时间（毫秒）。
+HTTP 请求体。只有目标 API 接口支持请求体时才需要配置。
### format [String]
-上游数据的格式，现在仅支持`json` `text`，默认是`json`。
-
-若你的数据格式为 `json`，需同时配置 schema 选项，例如：
-
-上游数据如下：
-
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
-
-您应该配置 schema 为以下内容：
-
-```hocon
+响应数据格式，支持 `json` 和 `text`。如果希望输出带字段名的数据行，请使用 `json` 并配置 `schema`。
-schema {
- fields {
- code = int
- data = string
- success = boolean
- }
-}
-
-```
-
-连接器将生成如下数据：
+### schema [Config]
-| code | data | success |
-|------|-------------|---------|
-| 200 | get success | true |
+当 `format = "json"` 时，用于定义输出行结构。更多信息请参考 [Schema
特性](../../introduction/concepts/schema-feature.md)。
-若你设置格式为 `text`，连接器不会对上游数据做出任何改变，示例：
+### json_field [Config]
-上游数据如下：
+把输出字段映射到 JSONPath 表达式。需要从嵌套 JSON 中取值时，可与 `schema` 一起使用。
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
+### content_field [String]
-连接器将生成如下数据：
+用于先截取 JSON 片段的 JSONPath 表达式，例如 `$.items[*]`。
-| content |
-|----------------------------------------------------------|
-| {"code": 200, "data": "get success", "success": true} |
+### pageing [Config]
-### schema [Config]
+继承自 HTTP 连接器的分页配置。任务配置中请保持 `pageing` 这个拼写。
-#### fields [Config]
+### poll_interval_millis [int]
-上游数据的字段定义。更多详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。
+流处理任务中的请求间隔，单位毫秒。批处理任务只读取一次后结束。
-### content_json [String]
+### retry [int]
-该参数可用于提取一些 json 数据。如果你只需要 “book” 部分的数据，可以配置 `content_field = "$.store.book.*"`.
+HTTP 请求因 `IOException` 失败时的最大重试次数。
-如果你的返回数据如下所示：
+### retry_backoff_multiplier_ms [int]
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+重试退避时间乘数，单位毫秒。
-你可以配置 `content_field = "$.store.book.*"` 并且结果返回如下：
-
-```json
-[
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
-]
-```
+### retry_backoff_max_ms [int]
-然后你可以通过更简单的 schema 配置获取所需的结果，例如：
+最大重试退避时间，单位毫秒。
-```hocon
-Http {
- url = "http://mockserver:1080/contentjson/mock";
- method = "GET"
- format = "json"
- content_field = "$.store.book.*"
- schema = {
- fields {
- category = string
- author = string
- title = string
- price = string
- }
- }
-}
-```
+### json_filed_missed_return_null [boolean]
-这是一个例子:
+设置为 `true` 时，JSON 字段缺失会返回 `null`；否则字段缺失会报错。
--
测试数据可参考此链接：[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
--
任务配置示例可参考此链接：[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+### common options
-### json_field [Config]
+源插件通用参数，请参考 [源通用选项](../common-options/source-common-options.md)。
-该参数用于帮助你配置 schema，因此必须与 schema 一起使用。
-
-如果你的数据如下所示：
-
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+## 示例
-你可以通过如下任务配置获取 “book” 部分的内容：
+读取 GitHub 组织下的仓库：
```hocon
source {
- Http {
- url = "http://mockserver:1080/jsonpath/mock";
+ Github {
+ url = "https://api.github.com/orgs/apache/repos";
+ access_token = "ghp_xxxxxxxxxxxx"
method = "GET"
format = "json"
- json_field = {
- category = "$.store.book[*].category"
- author = "$.store.book[*].author"
- title = "$.store.book[*].title"
- price = "$.store.book[*].price"
- }
schema = {
fields {
- category = string
- author = string
- title = string
- price = string
+ id = int
+ name = string
+ description = string
+ html_url = string
+ stargazers_count = int
+ forks = int
}
}
}
}
```
--
测试数据可参考此链接：[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
--
任务配置示例可参考此链接：[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
-
-### common options
-
-源插件通用参数，请参考 [常用选项](../common-options/source-common-options.md)获取详细说明。
-
-## 示例
+读取分页的 GitHub API 结果：
```hocon
-Github {
- url = "https://api.github.com/orgs/apache/repos";
- access_token = "xxxx"
- method = "GET"
- format = "json"
- schema = {
- fields {
- id = int
- name = string
- description = string
- html_url = string
- stargazers_count = int
- forks = int
+source {
+ Github {
+ url = "https://api.github.com/orgs/apache/repos";
+ access_token = "ghp_xxxxxxxxxxxx"
+ method = "GET"
+ params = {
+ per_page = "100"
+ page = "${page}"
+ }
+ pageing = {
+ page_field = "page"
+ total_page_size = 5
+ start_page_number = 1
+ use_placeholder_replacement = true
+ }
+ format = "json"
+ schema = {
+ fields {
+ id = int
+ name = string
+ html_url = string
+ }
}
}
}
@@ -296,4 +165,4 @@ Github {
## 变更日志
-<ChangeLog />
\ No newline at end of file
+<ChangeLog />
diff --git a/docs/zh/connectors/source/Gitlab.md
b/docs/zh/connectors/source/Gitlab.md
index 072dfe6f34..c609fd7857 100644
--- a/docs/zh/connectors/source/Gitlab.md
+++ b/docs/zh/connectors/source/Gitlab.md
@@ -6,294 +6,163 @@ import ChangeLog from
'../changelog/connector-http-gitlab.md';
## 描述
-用于从 Gitlab 读取数据。
+Gitlab 源连接器用于读取 GitLab REST API 数据。它基于 HTTP 源连接器实现，并会自动把 `access_token` 作为
GitLab `PRIVATE-TOKEN` 请求头发送。
-## 关键特性
+## 主要特性
-- [x] [批](../../introduction/concepts/connector-v2-features.md)
-- [ ] [流](../../introduction/concepts/connector-v2-features.md)
+- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
+- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
- [ ] [列投影](../../introduction/concepts/connector-v2-features.md)
-- [ ] [并行性](../../introduction/concepts/connector-v2-features.md)
-- [ ] [支持用户自定义split](../../introduction/concepts/connector-v2-features.md)
+- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
+- [ ] [支持用户自定义分片](../../introduction/concepts/connector-v2-features.md)
## 选项
-| 参数名 | 类型 | 必须 | 默认值 |
+| 参数名 | 类型 | 必填 | 默认值 |
|-----------------------------|---------|------|--------|
| url | String | 是 | - |
| access_token | String | 是 | - |
-| method | String | 否 | get |
-| schema.fields | Config | 否 | - |
-| format | String | 否 | json |
+| method | String | 否 | GET |
+| headers | Map | 否 | - |
| params | Map | 否 | - |
| body | String | 否 | - |
+| format | String | 否 | text |
+| schema | Config | 否 | - |
+| schema.fields | Config | 否 | - |
| json_field | Config | 否 | - |
-| content_json | String | 否 | - |
+| content_field | String | 否 | - |
+| pageing | Config | 否 | - |
| poll_interval_millis | int | 否 | - |
| retry | int | 否 | - |
| retry_backoff_multiplier_ms | int | 否 | 100 |
| retry_backoff_max_ms | int | 否 | 10000 |
-| enable_multi_lines | boolean | 否 | false |
+| json_filed_missed_return_null | boolean | 否 | false |
| common-options | config | 否 | - |
### url [String]
-http 请求 url
+GitLab REST API 地址，例如 `https://gitlab.com/api/v4/projects`。
### access_token [String]
-个人访问令牌
+GitLab 个人访问令牌。连接器会把它写入 HTTP `PRIVATE-TOKEN` 请求头。
### method [String]
-http 请求方法，仅支持 GET、POST 方法
-
-### params [Map]
-
-http 参数
-
-### body [String]
-
-http 请求体
-
-### poll_interval_millis [int]
-
-在流模式下请求 http api 的间隔（毫秒）
+HTTP 请求方法。常见的 GitLab 读取场景使用 `GET`。
-### retry [int]
+### headers [Map]
-如果 http 请求返回 `IOException` 的最大重试次数
+额外的 HTTP 请求头。除非你确实想覆盖由 `access_token` 生成的认证头，否则不要在这里配置 `PRIVATE-TOKEN`。
-### retry_backoff_multiplier_ms [int]
+### params [Map]
-如果 http 请求失败，重试退避时间（毫秒）乘数
+HTTP 查询参数，例如 `per_page`、`page`、`owned` 或其他 GitLab API 参数。
-### retry_backoff_max_ms [int]
+### body [String]
-如果 http 请求失败，最大重试退避时间（毫秒）
+HTTP 请求体。只有目标 API 接口支持请求体时才需要配置。
### format [String]
-上游数据的格式，现在仅支持 `json` `text`，默认 `json`。
-
-当您指定格式为 `json` 时，您还应该指定 schema 选项，例如：
-
-上游数据如下：
+响应数据格式，支持 `json` 和 `text`。如果希望输出带字段名的数据行，请使用 `json` 并配置 `schema`。
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
-
-您应该指定 schema 如下：
-
-```hocon
-
-schema {
- fields {
- code = int
- data = string
- success = boolean
- }
-}
-
-```
+### schema [Config]
-连接器将生成如下数据：
+当 `format = "json"` 时，用于定义输出行结构。更多信息请参考 [Schema
特性](../../introduction/concepts/schema-feature.md)。
-| code | data | success |
-|------|-------------|---------|
-| 200 | get success | true |
+### json_field [Config]
-当您指定格式为 `text` 时，连接器将对上游数据不做任何处理，例如：
+把输出字段映射到 JSONPath 表达式。需要从嵌套 JSON 中取值时，可与 `schema` 一起使用。
-上游数据如下：
+### content_field [String]
-```json
-{
- "code": 200,
- "data": "get success",
- "success": true
-}
-```
+用于先截取 JSON 片段的 JSONPath 表达式，例如 `$.items[*]`。
-连接器将生成如下数据：
+### pageing [Config]
-| content |
-|----------------------------------------------------------|
-| {"code": 200, "data": "get success", "success": true} |
+继承自 HTTP 连接器的分页配置。任务配置中请保持 `pageing` 这个拼写。
-### schema [Config]
-
-#### fields [Config]
+### poll_interval_millis [int]
-上游数据的模式字段。更多详情请参考 [Schema 特性](../../introduction/concepts/schema-feature.md)。
+该选项继承自 HTTP 连接器，但 Gitlab 源连接器当前只支持批处理模式。
-### content_json [String]
+### retry [int]
-此参数可以获取一些 json 数据。如果您只需要 'book' 部分中的数据，请配置 `content_field = "$.store.book.*"`。
+HTTP 请求因 `IOException` 失败时的最大重试次数。
-如果您的返回数据看起来像这样。
+### retry_backoff_multiplier_ms [int]
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+重试退避时间乘数，单位毫秒。
-您可以配置 `content_field = "$.store.book.*"`，返回的结果看起来像这样：
-
-```json
-[
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
-]
-```
+### retry_backoff_max_ms [int]
-然后您可以使用更简单的 schema 获得所需的结果，如
+最大重试退避时间，单位毫秒。
-```hocon
-Http {
- url = "http://mockserver:1080/contentjson/mock";
- method = "GET"
- format = "json"
- content_field = "$.store.book.*"
- schema = {
- fields {
- category = string
- author = string
- title = string
- price = string
- }
- }
-}
-```
+### json_filed_missed_return_null [boolean]
-这是一个示例：
+设置为 `true` 时，JSON 字段缺失会返回 `null`；否则字段缺失会报错。
-- 测试数据可以在此链接找到
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
-- 查看此链接了解任务配置
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf)。
+### common options
-### json_field [Config]
+源插件通用参数，请参考 [源通用选项](../common-options/source-common-options.md)。
-此参数可帮助您配置 schema，因此此参数必须与 schema 一起使用。
-
-如果您的数据看起来像这样：
-
-```json
-{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
-}
-```
+## 示例
-您可以通过配置任务如下来获取 'book' 的内容：
+读取 GitLab 项目：
```hocon
source {
- Http {
- url = "http://mockserver:1080/jsonpath/mock";
+ Gitlab {
+ url = "https://gitlab.com/api/v4/projects";
+ access_token = "glpat-xxxxxxxxxxxx"
method = "GET"
format = "json"
- json_field = {
- category = "$.store.book[*].category"
- author = "$.store.book[*].author"
- title = "$.store.book[*].title"
- price = "$.store.book[*].price"
- }
schema = {
fields {
- category = string
- author = string
- title = string
- price = string
+ id = int
+ description = string
+ name = string
+ name_with_namespace = string
+ path = string
+ http_url_to_repo = string
}
}
}
}
```
-- 测试数据可以在此链接找到
[mockserver-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json)
-- 查看此链接了解任务配置
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf)。
-
-### 通用选项
-
-源插件通用参数，请参考 [源通用选项](../common-options/source-common-options.md) 详见
-
-## 示例
+读取分页的 GitLab API 结果：
```hocon
-Gitlab{
+source {
+ Gitlab {
url = "https://gitlab.com/api/v4/projects";
- access_token = "xxxxx"
- schema {
- fields {
- id = int
- description = string
- name = string
- name_with_namespace = string
- path = string
- http_url_to_repo = string
- }
+ access_token = "glpat-xxxxxxxxxxxx"
+ method = "GET"
+ params = {
+ per_page = "100"
+ page = "${page}"
}
+ pageing = {
+ page_field = "page"
+ total_page_size = 5
+ start_page_number = 1
+ use_placeholder_replacement = true
+ }
+ format = "json"
+ schema = {
+ fields {
+ id = int
+ name = string
+ path = string
+ }
+ }
+ }
}
```
## 变更日志
<ChangeLog />
-
-
