This is an automated email from the ASF dual-hosted git repository.
gaojun2048 pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new 589e4161e [Improve][Connector-V2][Http]Improve json parse option rule
for all http connector (#3627)
589e4161e is described below
commit 589e4161ec810e1b8cd66b8e43314002a6244c06
Author: TaoZex <[email protected]>
AuthorDate: Wed Dec 7 17:32:07 2022 +0800
[Improve][Connector-V2][Http]Improve json parse option rule for all http
connector (#3627)
* [Improve][Connector-V2][Http]Improve json parse option rule for all http
connector
---
docs/en/connector-v2/source/Gitlab.md | 153 ++++++++++++++++++++-
docs/en/connector-v2/source/Http.md | 73 +++++-----
docs/en/connector-v2/source/Jira.md | 153 ++++++++++++++++++++-
docs/en/connector-v2/source/Klaviyo.md | 151 +++++++++++++++++++-
docs/en/connector-v2/source/Lemlist.md | 153 ++++++++++++++++++++-
docs/en/connector-v2/source/MyHours.md | 153 ++++++++++++++++++++-
docs/en/connector-v2/source/OneSignal.md | 153 ++++++++++++++++++++-
.../seatunnel/http/source/HttpSourceFactory.java | 29 ++--
.../gitlab/source/GitlabSourceFactory.java | 20 +--
.../seatunnel/jira/source/JiraSourceFactory.java | 26 +---
.../klaviyo/source/KlaviyoSourceFactory.java | 26 +---
.../lemlist/source/LemlistSourceFactory.java | 24 +---
.../myhours/source/MyHoursSourceFactory.java | 26 +---
.../onesignal/source/OneSignalSourceFactory.java | 24 +---
14 files changed, 965 insertions(+), 199 deletions(-)
diff --git a/docs/en/connector-v2/source/Gitlab.md
b/docs/en/connector-v2/source/Gitlab.md
index f907a93ee..d0b1df1bc 100644
--- a/docs/en/connector-v2/source/Gitlab.md
+++ b/docs/en/connector-v2/source/Gitlab.md
@@ -26,6 +26,8 @@ Used to read data from Gitlab.
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
+| json_field | Config | No | - |
+| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
@@ -77,9 +79,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
you should assign schema as the following:
@@ -107,9 +111,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
connector will generate data as the following:
@@ -124,6 +130,140 @@ connector will generate data as the following:
the schema fields of upstream data
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
+
+```json
+[
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+ url = "http://mockserver:1080/contentjson/mock";
+ method = "GET"
+ format = "json"
+ content_field = "$.store.book.*"
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used
with schema.
+
+If your data looks something like this:
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+ Http {
+ url = "http://mockserver:1080/jsonpath/mock";
+ method = "GET"
+ format = "json"
+ json_field = {
+ category = "$.store.book[*].category"
+ author = "$.store.book[*].author"
+ title = "$.store.book[*].title"
+ price = "$.store.book[*].price"
+ }
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+ }
+}
+```
+
+- Test data can be found at this link
[mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
@@ -152,3 +292,4 @@ Gitlab{
### next version
- Add Gitlab Source Connector
+- [Feature][Connector-V2][HTTP] Use json-path parsing
([3510](https://github.com/apache/incubator-seatunnel/pull/3510))
diff --git a/docs/en/connector-v2/source/Http.md
b/docs/en/connector-v2/source/Http.md
index 045a70693..6e0ceb281 100644
--- a/docs/en/connector-v2/source/Http.md
+++ b/docs/en/connector-v2/source/Http.md
@@ -80,13 +80,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-
{
"code": 200,
"data": "get success",
"success": true
}
-
```
you should assign schema as the following:
@@ -114,13 +112,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-
{
"code": 200,
"data": "get success",
"success": true
}
-
```
connector will generate data as the following:
@@ -143,46 +139,46 @@ If your return data looks something like this.
```json
{
- "store": {
- "book": [
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ],
- "bicycle": {
- "color": "red",
- "price": 19.95
- }
- },
- "expensive": 10
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
}
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
```
You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
```json
[
- {
- "category": "reference",
- "author": "Nigel Rees",
- "title": "Sayings of the Century",
- "price": 8.95
- },
- {
- "category": "fiction",
- "author": "Evelyn Waugh",
- "title": "Sword of Honour",
- "price": 12.99
- }
- ]
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
```
Then you can get the desired result with a simpler schema,like
@@ -208,7 +204,6 @@ Here is an example:
- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
-
### json_field [Config]
This parameter helps you configure the schema,so this parameter must be used
with schema.
diff --git a/docs/en/connector-v2/source/Jira.md
b/docs/en/connector-v2/source/Jira.md
index 0f170b695..1614b783c 100644
--- a/docs/en/connector-v2/source/Jira.md
+++ b/docs/en/connector-v2/source/Jira.md
@@ -27,6 +27,8 @@ Used to read data from Jira.
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
+| json_field | Config | No | - |
+| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
@@ -84,9 +86,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
you should assign schema as the following:
@@ -114,9 +118,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
connector will generate data as the following:
@@ -131,6 +137,140 @@ connector will generate data as the following:
the schema fields of upstream data
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
+
+```json
+[
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+ url = "http://mockserver:1080/contentjson/mock";
+ method = "GET"
+ format = "json"
+ content_field = "$.store.book.*"
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used
with schema.
+
+If your data looks something like this:
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+ Http {
+ url = "http://mockserver:1080/jsonpath/mock";
+ method = "GET"
+ format = "json"
+ json_field = {
+ category = "$.store.book[*].category"
+ author = "$.store.book[*].author"
+ title = "$.store.book[*].title"
+ price = "$.store.book[*].price"
+ }
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+ }
+}
+```
+
+- Test data can be found at this link
[mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
@@ -158,3 +298,4 @@ Jira {
### next version
- Add Jira Source Connector
+- [Feature][Connector-V2][HTTP] Use json-path parsing
([3510](https://github.com/apache/incubator-seatunnel/pull/3510))
\ No newline at end of file
diff --git a/docs/en/connector-v2/source/Klaviyo.md
b/docs/en/connector-v2/source/Klaviyo.md
index 6880b4f1d..242f08b68 100644
--- a/docs/en/connector-v2/source/Klaviyo.md
+++ b/docs/en/connector-v2/source/Klaviyo.md
@@ -28,6 +28,8 @@ Used to read data from Klaviyo.
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
+| json_field | Config | No | - |
+| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
@@ -85,7 +87,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-{"code": 200, "data": "get success", "success": true}
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
you should assign schema as the following:
@@ -111,7 +117,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-{"code": 200, "data": "get success", "success": true}
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
connector will generate data as the following:
@@ -126,6 +136,140 @@ connector will generate data as the following:
the schema fields of upstream data
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
+
+```json
+[
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+ url = "http://mockserver:1080/contentjson/mock";
+ method = "GET"
+ format = "json"
+ content_field = "$.store.book.*"
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used
with schema.
+
+If your data looks something like this:
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+ Http {
+ url = "http://mockserver:1080/jsonpath/mock";
+ method = "GET"
+ format = "json"
+ json_field = {
+ category = "$.store.book[*].category"
+ author = "$.store.book[*].author"
+ title = "$.store.book[*].title"
+ price = "$.store.book[*].price"
+ }
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+ }
+}
+```
+
+- Test data can be found at this link
[mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
@@ -160,4 +304,5 @@ Klaviyo {
### next version
-- Add Klaviyo Source Connector
\ No newline at end of file
+- Add Klaviyo Source Connector
+- [Feature][Connector-V2][HTTP] Use json-path parsing
([3510](https://github.com/apache/incubator-seatunnel/pull/3510))
\ No newline at end of file
diff --git a/docs/en/connector-v2/source/Lemlist.md
b/docs/en/connector-v2/source/Lemlist.md
index 8b21bce5d..f5669ac6f 100644
--- a/docs/en/connector-v2/source/Lemlist.md
+++ b/docs/en/connector-v2/source/Lemlist.md
@@ -26,6 +26,8 @@ Used to read data from Lemlist.
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
+| json_field | Config | No | - |
+| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
@@ -79,9 +81,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
you should assign schema as the following:
@@ -109,9 +113,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
connector will generate data as the following:
@@ -126,6 +132,140 @@ connector will generate data as the following:
the schema fields of upstream data
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
+
+```json
+[
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+ url = "http://mockserver:1080/contentjson/mock";
+ method = "GET"
+ format = "json"
+ content_field = "$.store.book.*"
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used
with schema.
+
+If your data looks something like this:
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+ Http {
+ url = "http://mockserver:1080/jsonpath/mock";
+ method = "GET"
+ format = "json"
+ json_field = {
+ category = "$.store.book[*].category"
+ author = "$.store.book[*].author"
+ title = "$.store.book[*].title"
+ price = "$.store.book[*].price"
+ }
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+ }
+}
+```
+
+- Test data can be found at this link
[mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
@@ -150,3 +290,4 @@ Lemlist {
### next version
- Add Lemlist Source Connector
+- [Feature][Connector-V2][HTTP] Use json-path parsing
([3510](https://github.com/apache/incubator-seatunnel/pull/3510))
diff --git a/docs/en/connector-v2/source/MyHours.md
b/docs/en/connector-v2/source/MyHours.md
index c5f5a0bb4..de02a030f 100644
--- a/docs/en/connector-v2/source/MyHours.md
+++ b/docs/en/connector-v2/source/MyHours.md
@@ -28,6 +28,8 @@ Used to read data from My Hours.
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
+| json_field | Config | No | - |
+| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
@@ -83,9 +85,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
you should assign schema as the following:
@@ -113,9 +117,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
connector will generate data as the following:
@@ -130,6 +136,140 @@ connector will generate data as the following:
the schema fields of upstream data
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
+
+```json
+[
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+ url = "http://mockserver:1080/contentjson/mock";
+ method = "GET"
+ format = "json"
+ content_field = "$.store.book.*"
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used
with schema.
+
+If your data looks something like this:
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+ Http {
+ url = "http://mockserver:1080/jsonpath/mock";
+ method = "GET"
+ format = "json"
+ json_field = {
+ category = "$.store.book[*].category"
+ author = "$.store.book[*].author"
+ title = "$.store.book[*].title"
+ price = "$.store.book[*].price"
+ }
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+ }
+}
+```
+
+- Test data can be found at this link
[mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
@@ -176,3 +316,4 @@ MyHours{
### next version
- Add My Hours Source Connector
+- [Feature][Connector-V2][HTTP] Use json-path parsing
([3510](https://github.com/apache/incubator-seatunnel/pull/3510))
diff --git a/docs/en/connector-v2/source/OneSignal.md
b/docs/en/connector-v2/source/OneSignal.md
index 9e0286537..852fc75ff 100644
--- a/docs/en/connector-v2/source/OneSignal.md
+++ b/docs/en/connector-v2/source/OneSignal.md
@@ -27,6 +27,8 @@ Used to read data from OneSignal.
| format | String | No | json |
| params | Map | No | - |
| body | String | No | - |
+| json_field | Config | No | - |
+| content_json | String | No | - |
| poll_interval_ms | int | No | - |
| retry | int | No | - |
| retry_backoff_multiplier_ms | int | No | 100 |
@@ -80,9 +82,11 @@ when you assign format is `json`, you should also assign
schema option, for exam
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
you should assign schema as the following:
@@ -110,9 +114,11 @@ when you assign format is `text`, connector will do
nothing for upstream data, f
upstream data is the following:
```json
-
-{"code": 200, "data": "get success", "success": true}
-
+{
+ "code": 200,
+ "data": "get success",
+ "success": true
+}
```
connector will generate data as the following:
@@ -127,6 +133,140 @@ connector will generate data as the following:
the schema fields of upstream data
+### content_json [String]
+
+This parameter can get some json data.If you only need the data in the 'book'
section, configure `content_field = "$.store.book.*"`.
+
+If your return data looks something like this.
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+You can configure `content_field = "$.store.book.*"` and the result returned
looks like this:
+
+```json
+[
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+]
+```
+Then you can get the desired result with a simpler schema,like
+
+```hocon
+Http {
+ url = "http://mockserver:1080/contentjson/mock";
+ method = "GET"
+ format = "json"
+ content_field = "$.store.book.*"
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+}
+```
+
+Here is an example:
+
+- Test data can be found at this link
[mockserver-contentjson-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-contentjson-config.json)
+- See this link for task configuration
[http_contentjson_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_contentjson_to_assert.conf).
+
+### json_field [Config]
+
+This parameter helps you configure the schema,so this parameter must be used
with schema.
+
+If your data looks something like this:
+
+```json
+{
+ "store": {
+ "book": [
+ {
+ "category": "reference",
+ "author": "Nigel Rees",
+ "title": "Sayings of the Century",
+ "price": 8.95
+ },
+ {
+ "category": "fiction",
+ "author": "Evelyn Waugh",
+ "title": "Sword of Honour",
+ "price": 12.99
+ }
+ ],
+ "bicycle": {
+ "color": "red",
+ "price": 19.95
+ }
+ },
+ "expensive": 10
+}
+```
+
+You can get the contents of 'book' by configuring the task as follows:
+
+```hocon
+source {
+ Http {
+ url = "http://mockserver:1080/jsonpath/mock";
+ method = "GET"
+ format = "json"
+ json_field = {
+ category = "$.store.book[*].category"
+ author = "$.store.book[*].author"
+ title = "$.store.book[*].title"
+ price = "$.store.book[*].price"
+ }
+ schema = {
+ fields {
+ category = string
+ author = string
+ title = string
+ price = string
+ }
+ }
+ }
+}
+```
+
+- Test data can be found at this link
[mockserver-jsonpath-config.json](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-jsonpath-config.json)
+- See this link for task configuration
[http_jsonpath_to_assert.conf](../../../../seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/http_jsonpath_to_assert.conf).
+
### common options
Source plugin common parameters, please refer to [Source Common
Options](common-options.md) for details
@@ -180,3 +320,4 @@ OneSignal {
### next version
- Add OneSignal Source Connector
+- [Feature][Connector-V2][HTTP] Use json-path parsing
([3510](https://github.com/apache/incubator-seatunnel/pull/3510))
\ No newline at end of file
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceFactory.java
index 49a5f1eb3..7c89f20cb 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/http/source/HttpSourceFactory.java
@@ -36,18 +36,23 @@ public class HttpSourceFactory implements
TableSourceFactory {
@Override
public OptionRule optionRule() {
+ return getHttpBuilder().build();
+ }
+
+ public OptionRule.Builder getHttpBuilder() {
return OptionRule.builder()
- .required(HttpConfig.URL)
- .optional(HttpConfig.METHOD)
- .optional(HttpConfig.HEADERS)
- .optional(HttpConfig.PARAMS)
- .optional(HttpConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
HttpConfig.BODY)
- .conditional(HttpConfig.FORMAT, HttpConfig.ResponseFormat.JSON,
SeaTunnelSchema.SCHEMA)
- .optional(HttpConfig.POLL_INTERVAL_MILLS)
- .optional(HttpConfig.RETRY)
- .optional(HttpConfig.RETRY_BACKOFF_MULTIPLIER_MS)
- .optional(HttpConfig.RETRY_BACKOFF_MAX_MS)
- .build();
+ .required(HttpConfig.URL)
+ .optional(HttpConfig.METHOD)
+ .optional(HttpConfig.HEADERS)
+ .optional(HttpConfig.PARAMS)
+ .optional(HttpConfig.FORMAT)
+ .optional(HttpConfig.JSON_FIELD)
+ .optional(HttpConfig.CONTENT_FIELD)
+ .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
HttpConfig.BODY)
+ .conditional(HttpConfig.FORMAT,
HttpConfig.ResponseFormat.JSON, SeaTunnelSchema.SCHEMA)
+ .optional(HttpConfig.POLL_INTERVAL_MILLS)
+ .optional(HttpConfig.RETRY)
+ .optional(HttpConfig.RETRY_BACKOFF_MULTIPLIER_MS)
+ .optional(HttpConfig.RETRY_BACKOFF_MAX_MS);
}
}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-gitlab/src/main/java/org/apache/seatunnel/connectors/seatunnel/gitlab/source/GitlabSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-gitlab/src/main/java/org/apache/seatunnel/connectors/seatunnel/gitlab/source/GitlabSourceFactory.java
index 76423de5d..3a04c663d 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-gitlab/src/main/java/org/apache/seatunnel/connectors/seatunnel/gitlab/source/GitlabSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-gitlab/src/main/java/org/apache/seatunnel/connectors/seatunnel/gitlab/source/GitlabSourceFactory.java
@@ -19,16 +19,13 @@ package
org.apache.seatunnel.connectors.seatunnel.gitlab.source;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.table.factory.Factory;
-import org.apache.seatunnel.api.table.factory.TableSourceFactory;
-import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
import
org.apache.seatunnel.connectors.seatunnel.gitlab.source.config.GitlabSourceConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
import com.google.auto.service.AutoService;
@AutoService(Factory.class)
-public class GitlabSourceFactory implements TableSourceFactory {
+public class GitlabSourceFactory extends HttpSourceFactory {
@Override
public String factoryIdentifier() {
return "Gitlab";
@@ -36,19 +33,8 @@ public class GitlabSourceFactory implements
TableSourceFactory {
@Override
public OptionRule optionRule() {
- return OptionRule.builder()
- .required(GitlabSourceConfig.URL)
+ return getHttpBuilder()
.required(GitlabSourceConfig.ACCESS_TOKEN)
- .optional(GitlabSourceConfig.METHOD)
- .optional(GitlabSourceConfig.HEADERS)
- .optional(GitlabSourceConfig.PARAMS)
- .optional(GitlabSourceConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
GitlabSourceConfig.BODY)
- .conditional(HttpConfig.FORMAT,
HttpConfig.ResponseFormat.JSON, SeaTunnelSchema.SCHEMA)
- .optional(GitlabSourceConfig.POLL_INTERVAL_MILLS)
- .optional(GitlabSourceConfig.RETRY)
- .optional(GitlabSourceConfig.RETRY_BACKOFF_MAX_MS)
- .optional(GitlabSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
.build();
}
}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java
index e623c989c..7c809ea42 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-jira/src/main/java/org/apache/seatunnel/connectors/seatunnel/jira/source/JiraSourceFactory.java
@@ -19,16 +19,13 @@ package
org.apache.seatunnel.connectors.seatunnel.jira.source;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.table.factory.Factory;
-import org.apache.seatunnel.api.table.factory.TableSourceFactory;
-import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
import
org.apache.seatunnel.connectors.seatunnel.jira.source.config.JiraSourceConfig;
import com.google.auto.service.AutoService;
@AutoService(Factory.class)
-public class JiraSourceFactory implements TableSourceFactory {
+public class JiraSourceFactory extends HttpSourceFactory {
@Override
public String factoryIdentifier() {
return "Jira";
@@ -36,20 +33,9 @@ public class JiraSourceFactory implements TableSourceFactory
{
@Override
public OptionRule optionRule() {
- return OptionRule.builder()
- .required(JiraSourceConfig.URL)
- .required(JiraSourceConfig.EMAIL)
- .required(JiraSourceConfig.API_TOKEN)
- .optional(JiraSourceConfig.METHOD)
- .optional(JiraSourceConfig.HEADERS)
- .optional(JiraSourceConfig.PARAMS)
- .optional(JiraSourceConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
JiraSourceConfig.BODY)
- .conditional(HttpConfig.FORMAT, HttpConfig.ResponseFormat.JSON,
SeaTunnelSchema.SCHEMA)
- .optional(JiraSourceConfig.POLL_INTERVAL_MILLS)
- .optional(JiraSourceConfig.RETRY)
- .optional(JiraSourceConfig.RETRY_BACKOFF_MAX_MS)
- .optional(JiraSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
- .build();
+ return getHttpBuilder()
+ .required(JiraSourceConfig.EMAIL)
+ .required(JiraSourceConfig.API_TOKEN)
+ .build();
}
}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-klaviyo/src/main/java/org/apache/seatunnel/connectors/seatunnel/klaviyo/source/KlaviyoSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-klaviyo/src/main/java/org/apache/seatunnel/connectors/seatunnel/klaviyo/source/KlaviyoSourceFactory.java
index fff5633bd..4b2817122 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-klaviyo/src/main/java/org/apache/seatunnel/connectors/seatunnel/klaviyo/source/KlaviyoSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-klaviyo/src/main/java/org/apache/seatunnel/connectors/seatunnel/klaviyo/source/KlaviyoSourceFactory.java
@@ -19,16 +19,13 @@ package
org.apache.seatunnel.connectors.seatunnel.klaviyo.source;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.table.factory.Factory;
-import org.apache.seatunnel.api.table.factory.TableSourceFactory;
-import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
import
org.apache.seatunnel.connectors.seatunnel.klaviyo.source.config.KlaviyoSourceConfig;
import com.google.auto.service.AutoService;
@AutoService(Factory.class)
-public class KlaviyoSourceFactory implements TableSourceFactory {
+public class KlaviyoSourceFactory extends HttpSourceFactory {
@Override
public String factoryIdentifier() {
return "Klaviyo";
@@ -36,20 +33,9 @@ public class KlaviyoSourceFactory implements
TableSourceFactory {
@Override
public OptionRule optionRule() {
- return OptionRule.builder()
- .required(KlaviyoSourceConfig.URL)
- .required(KlaviyoSourceConfig.PRIVATE_KEY)
- .required(KlaviyoSourceConfig.REVISION)
- .optional(KlaviyoSourceConfig.METHOD)
- .optional(KlaviyoSourceConfig.HEADERS)
- .optional(KlaviyoSourceConfig.PARAMS)
- .optional(KlaviyoSourceConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
KlaviyoSourceConfig.BODY)
- .conditional(HttpConfig.FORMAT, HttpConfig.ResponseFormat.JSON,
SeaTunnelSchema.SCHEMA)
- .optional(KlaviyoSourceConfig.POLL_INTERVAL_MILLS)
- .optional(KlaviyoSourceConfig.RETRY)
- .optional(KlaviyoSourceConfig.RETRY_BACKOFF_MAX_MS)
- .optional(KlaviyoSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
- .build();
+ return getHttpBuilder()
+ .required(KlaviyoSourceConfig.PRIVATE_KEY)
+ .required(KlaviyoSourceConfig.REVISION)
+ .build();
}
}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSourceFactory.java
index eaaad2e70..2c5390fa4 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-lemlist/src/main/java/org/apache/seatunnel/connectors/seatunnel/lemlist/source/LemlistSourceFactory.java
@@ -19,16 +19,13 @@ package
org.apache.seatunnel.connectors.seatunnel.lemlist.source;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.table.factory.Factory;
-import org.apache.seatunnel.api.table.factory.TableSourceFactory;
-import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
import
org.apache.seatunnel.connectors.seatunnel.lemlist.source.config.LemlistSourceConfig;
import com.google.auto.service.AutoService;
@AutoService(Factory.class)
-public class LemlistSourceFactory implements TableSourceFactory {
+public class LemlistSourceFactory extends HttpSourceFactory {
@Override
public String factoryIdentifier() {
return "Lemlist";
@@ -36,19 +33,8 @@ public class LemlistSourceFactory implements
TableSourceFactory {
@Override
public OptionRule optionRule() {
- return OptionRule.builder()
- .required(LemlistSourceConfig.URL)
- .required(LemlistSourceConfig.PASSWORD)
- .optional(LemlistSourceConfig.METHOD)
- .optional(LemlistSourceConfig.HEADERS)
- .optional(LemlistSourceConfig.PARAMS)
- .optional(LemlistSourceConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
LemlistSourceConfig.BODY)
- .conditional(HttpConfig.FORMAT, HttpConfig.ResponseFormat.JSON,
SeaTunnelSchema.SCHEMA)
- .optional(LemlistSourceConfig.POLL_INTERVAL_MILLS)
- .optional(LemlistSourceConfig.RETRY)
- .optional(LemlistSourceConfig.RETRY_BACKOFF_MAX_MS)
- .optional(LemlistSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
- .build();
+ return getHttpBuilder()
+ .required(LemlistSourceConfig.PASSWORD)
+ .build();
}
}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-myhours/src/main/java/org/apache/seatunnel/connectors/seatunnel/myhours/source/MyHoursSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-myhours/src/main/java/org/apache/seatunnel/connectors/seatunnel/myhours/source/MyHoursSourceFactory.java
index f476e0f04..24daeba1b 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-myhours/src/main/java/org/apache/seatunnel/connectors/seatunnel/myhours/source/MyHoursSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-myhours/src/main/java/org/apache/seatunnel/connectors/seatunnel/myhours/source/MyHoursSourceFactory.java
@@ -19,16 +19,13 @@ package
org.apache.seatunnel.connectors.seatunnel.myhours.source;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.table.factory.Factory;
-import org.apache.seatunnel.api.table.factory.TableSourceFactory;
-import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
import
org.apache.seatunnel.connectors.seatunnel.myhours.source.config.MyHoursSourceConfig;
import com.google.auto.service.AutoService;
@AutoService(Factory.class)
-public class MyHoursSourceFactory implements TableSourceFactory {
+public class MyHoursSourceFactory extends HttpSourceFactory {
@Override
public String factoryIdentifier() {
return "MyHours";
@@ -36,20 +33,9 @@ public class MyHoursSourceFactory implements
TableSourceFactory {
@Override
public OptionRule optionRule() {
- return OptionRule.builder()
- .required(MyHoursSourceConfig.URL)
- .required(MyHoursSourceConfig.EMAIL)
- .required(MyHoursSourceConfig.PASSWORD)
- .optional(MyHoursSourceConfig.METHOD)
- .optional(MyHoursSourceConfig.HEADERS)
- .optional(MyHoursSourceConfig.PARAMS)
- .optional(MyHoursSourceConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
MyHoursSourceConfig.BODY)
- .conditional(HttpConfig.FORMAT, HttpConfig.ResponseFormat.JSON,
SeaTunnelSchema.SCHEMA)
- .optional(MyHoursSourceConfig.POLL_INTERVAL_MILLS)
- .optional(MyHoursSourceConfig.RETRY)
- .optional(MyHoursSourceConfig.RETRY_BACKOFF_MAX_MS)
- .optional(MyHoursSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
- .build();
+ return getHttpBuilder()
+ .required(MyHoursSourceConfig.EMAIL)
+ .required(MyHoursSourceConfig.PASSWORD)
+ .build();
}
}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-onesignal/src/main/java/org/apache/seatunnel/connectors/seatunnel/onesignal/source/OneSignalSourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-onesignal/src/main/java/org/apache/seatunnel/connectors/seatunnel/onesignal/source/OneSignalSourceFactory.java
index c6c83afe6..acbff75f6 100644
---
a/seatunnel-connectors-v2/connector-http/connector-http-onesignal/src/main/java/org/apache/seatunnel/connectors/seatunnel/onesignal/source/OneSignalSourceFactory.java
+++
b/seatunnel-connectors-v2/connector-http/connector-http-onesignal/src/main/java/org/apache/seatunnel/connectors/seatunnel/onesignal/source/OneSignalSourceFactory.java
@@ -19,16 +19,13 @@ package
org.apache.seatunnel.connectors.seatunnel.onesignal.source;
import org.apache.seatunnel.api.configuration.util.OptionRule;
import org.apache.seatunnel.api.table.factory.Factory;
-import org.apache.seatunnel.api.table.factory.TableSourceFactory;
-import org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpConfig;
-import org.apache.seatunnel.connectors.seatunnel.http.config.HttpRequestMethod;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
import
org.apache.seatunnel.connectors.seatunnel.onesignal.source.config.OneSignalSourceConfig;
import com.google.auto.service.AutoService;
@AutoService(Factory.class)
-public class OneSignalSourceFactory implements TableSourceFactory {
+public class OneSignalSourceFactory extends HttpSourceFactory {
@Override
public String factoryIdentifier() {
return "OneSignal";
@@ -36,19 +33,8 @@ public class OneSignalSourceFactory implements
TableSourceFactory {
@Override
public OptionRule optionRule() {
- return OptionRule.builder()
- .required(OneSignalSourceConfig.URL)
- .required(OneSignalSourceConfig.PASSWORD)
- .optional(OneSignalSourceConfig.METHOD)
- .optional(OneSignalSourceConfig.HEADERS)
- .optional(OneSignalSourceConfig.PARAMS)
- .optional(OneSignalSourceConfig.FORMAT)
- .conditional(HttpConfig.METHOD, HttpRequestMethod.POST,
OneSignalSourceConfig.BODY)
- .conditional(HttpConfig.FORMAT, HttpConfig.ResponseFormat.JSON,
SeaTunnelSchema.SCHEMA)
- .optional(OneSignalSourceConfig.POLL_INTERVAL_MILLS)
- .optional(OneSignalSourceConfig.RETRY)
- .optional(OneSignalSourceConfig.RETRY_BACKOFF_MAX_MS)
- .optional(OneSignalSourceConfig.RETRY_BACKOFF_MULTIPLIER_MS)
- .build();
+ return getHttpBuilder()
+ .required(OneSignalSourceConfig.PASSWORD)
+ .build();
}
}
