This is an automated email from the ASF dual-hosted git repository.
github-merge-queue[bot] pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new b05f17c317 [Feature][Connector-V2][Shopify] Add Shopify source
connector (#11028)
b05f17c317 is described below
commit b05f17c3179225ba6095185ad80a52a7aaa483cd
Author: Gabriel Baldez <[email protected]>
AuthorDate: Sat Sep 5 05:17:25 2026 +0000
[Feature][Connector-V2][Shopify] Add Shopify source connector (#11028)
Co-authored-by: Daniel Leens <[email protected]>
---
.../connectors/changelog/connector-http-shopify.md | 6 +
docs/en/connectors/source/Shopify.md | 135 +++++++++++++++++++++
.../connectors/changelog/connector-http-shopify.md | 6 +
docs/zh/connectors/source/Shopify.md | 131 ++++++++++++++++++++
plugin-mapping.properties | 1 +
.../{ => connector-http-shopify}/pom.xml | 33 ++---
.../seatunnel/shopify/source/ShopifySource.java | 87 +++++++++++++
.../shopify/source/ShopifySourceFactory.java | 64 ++++++++++
.../source/config/ShopifySourceOptions.java | 38 ++++++
.../source/config/ShopifySourceParameter.java | 43 +++++++
.../seatunnel/shopify/ShopifyFactoryTest.java | 31 +++++
.../shopify/ShopifySourceParameterTest.java | 52 ++++++++
.../seatunnel/shopify/ShopifySourceTest.java | 66 ++++++++++
seatunnel-connectors-v2/connector-http/pom.xml | 1 +
seatunnel-dist/pom.xml | 6 +
.../connector-http-e2e/pom.xml | 6 +
.../seatunnel/e2e/connector/http/HttpIT.java | 4 +
.../src/test/resources/mockserver-config.json | 57 +++++++++
.../src/test/resources/shopify_json_to_assert.conf | 97 +++++++++++++++
19 files changed, 842 insertions(+), 22 deletions(-)
diff --git a/docs/en/connectors/changelog/connector-http-shopify.md
b/docs/en/connectors/changelog/connector-http-shopify.md
new file mode 100644
index 0000000000..1cf41e674b
--- /dev/null
+++ b/docs/en/connectors/changelog/connector-http-shopify.md
@@ -0,0 +1,6 @@
+<details><summary> Change Log </summary>
+
+| Change | Commit | Version |
+| --- | --- | --- |
+
+</details>
diff --git a/docs/en/connectors/source/Shopify.md
b/docs/en/connectors/source/Shopify.md
new file mode 100644
index 0000000000..744f191007
--- /dev/null
+++ b/docs/en/connectors/source/Shopify.md
@@ -0,0 +1,135 @@
+import ChangeLog from '../changelog/connector-http-shopify.md';
+
+# Shopify
+
+> Shopify source connector
+
+## Description
+
+Used to read data from the [Shopify Admin REST
API](https://shopify.dev/docs/api/admin-rest). It authenticates with a Shopify
Admin API access token (sent in the `X-Shopify-Access-Token` header) and reads
a resource endpoint such as orders, products, or customers into SeaTunnel rows.
+
+## Key features
+
+- [x] [batch](../../introduction/concepts/connector-v2-features.md)
+- [ ] [stream](../../introduction/concepts/connector-v2-features.md)
+- [ ] [exactly-once](../../introduction/concepts/connector-v2-features.md)
+- [x] [column projection](../../introduction/concepts/connector-v2-features.md)
+- [ ] [parallelism](../../introduction/concepts/connector-v2-features.md)
+- [ ] [support user-defined
split](../../introduction/concepts/connector-v2-features.md)
+
+## Options
+
+| name | type | required | default value |
+|-----------------------------|---------|----------|---------------|
+| url | String | Yes | - |
+| access_token | String | Yes | - |
+| method | String | No | get |
+| headers | Map | No | - |
+| schema | Config | No | - |
+| format | String | No | text |
+| params | Map | No | - |
+| body | String | No | - |
+| json_field | Config | No | - |
+| content_field | String | No | - |
+| poll_interval_millis | int | No | - |
+| retry | int | No | - |
+| retry_backoff_multiplier_ms | int | No | 100 |
+| retry_backoff_max_ms | int | No | 10000 |
+| json_filed_missed_return_null | boolean | No | false |
+| enable_multi_lines | boolean | No | false |
+| common-options | config | No | - |
+
+`pageing` appears in the option rule but is rejected at startup by this
connector — see [Pagination](#pagination).
+
+### url [String]
+
+The Shopify Admin API endpoint to read from, for example
`https://your-store.myshopify.com/admin/api/2024-01/products.json`.
+
+### access_token [String]
+
+The Shopify Admin API access token. It is sent in the `X-Shopify-Access-Token`
request header. See the [Shopify authentication
docs](https://shopify.dev/docs/api/admin-rest#authentication) for how to obtain
one.
+
+### method [String]
+
+http request method, only supports GET, POST method.
+
+### headers [Map]
+
+Extra HTTP request headers. The connector already sets
`X-Shopify-Access-Token` from
+`access_token` and `Accept: application/json`, so this is only for anything
beyond those —
+setting `X-Shopify-Access-Token` here is overwritten by `access_token`.
+
+### schema [Config]
+
+The structure of the data, including field names and field types. For more
details, please refer to [Schema
Feature](../../introduction/concepts/schema-feature.md).
+
+### format [String]
+
+the format of upstream data, now only support `json` `text`, default `text`.
+
+### params [Map]
+
+http params
+
+### json_field [Config]
+
+This parameter helps you configure the schema, so this parameter must be used
with schema. It maps JSON paths in the response to schema fields. See the [Http
source](./Http.md) connector for details and examples.
+
+### content_field [String]
+
+This parameter can extract a sub-section of the JSON response (for example the
array under a top-level key such as `products` or `orders`) before mapping to
rows. See the [Http source](./Http.md) connector for details and examples.
+
+### common options
+
+Source plugin common parameters, please refer to [Source Common
Options](../common-options/source-common-options.md) for details.
+
+## Pagination
+
+**Not supported.** `pageing` is inherited from the HTTP source option rule,
but this connector
+does not pass it to the reader, so honouring it would read only the first
response while the job
+reported success. Setting it therefore fails at startup with `HTTP-03`, rather
than silently
+returning partial data.
+
+Wiring the inherited pagination through would not help on its own: the shared
implementation
+reads the next cursor out of the response *body* with a JsonPath, while the
Admin REST API
+returns it in the `Link` response header. Supporting it properly means teaching
+`connector-http-base` to read a header cursor.
+
+## Example
+
+```hocon
+source {
+ Shopify {
+ url = "https://your-store.myshopify.com/admin/api/2024-01/products.json";
+ access_token = "${SHOPIFY_ACCESS_TOKEN}"
+ method = "GET"
+ format = "json"
+ content_field = "$.products.*"
+ schema = {
+ fields {
+ id = string
+ title = string
+ vendor = string
+ product_type = string
+ created_at = string
+ updated_at = string
+ }
+ }
+ }
+}
+```
+
+`${SHOPIFY_ACCESS_TOKEN}` is a SeaTunnel config variable, not an environment
variable — it is
+substituted only when the value is supplied on the command line:
+
+```bash
+./bin/seatunnel.sh -c your_app.conf -i SHOPIFY_ACCESS_TOKEN=shpat_xxx
+```
+
+Without `-i`, the literal text `${SHOPIFY_ACCESS_TOKEN}` is sent as the token
and Shopify answers
+`401`. See [variable configuration](../../introduction/concepts/config.md).
+
+
+## Changelog
+
+<ChangeLog />
diff --git a/docs/zh/connectors/changelog/connector-http-shopify.md
b/docs/zh/connectors/changelog/connector-http-shopify.md
new file mode 100644
index 0000000000..1cf41e674b
--- /dev/null
+++ b/docs/zh/connectors/changelog/connector-http-shopify.md
@@ -0,0 +1,6 @@
+<details><summary> Change Log </summary>
+
+| Change | Commit | Version |
+| --- | --- | --- |
+
+</details>
diff --git a/docs/zh/connectors/source/Shopify.md
b/docs/zh/connectors/source/Shopify.md
new file mode 100644
index 0000000000..adf9181ec5
--- /dev/null
+++ b/docs/zh/connectors/source/Shopify.md
@@ -0,0 +1,131 @@
+import ChangeLog from '../changelog/connector-http-shopify.md';
+
+# Shopify
+
+> Shopify 数据源连接器
+
+## 描述
+
+用于从 [Shopify Admin REST API](https://shopify.dev/docs/api/admin-rest) 读取数据。它使用
Shopify Admin API 的访问令牌进行认证（通过 `X-Shopify-Access-Token` 请求头发送），并将某个资源接口（如
orders、products、customers）读取为 SeaTunnel 的行数据。
+
+## 主要特性
+
+- [x] [批处理](../../introduction/concepts/connector-v2-features.md)
+- [ ] [流处理](../../introduction/concepts/connector-v2-features.md)
+- [ ] [精确一次](../../introduction/concepts/connector-v2-features.md)
+- [x] [列投影](../../introduction/concepts/connector-v2-features.md)
+- [ ] [并行度](../../introduction/concepts/connector-v2-features.md)
+- [ ] [支持用户自定义分片](../../introduction/concepts/connector-v2-features.md)
+
+## 选项
+
+| 名称 | 类型 | 是否必填 | 默认值 |
+|-----------------------------|---------|----------|-----------|
+| url | String | 是 | - |
+| access_token | String | 是 | - |
+| method | String | 否 | get |
+| headers | Map | 否 | - |
+| schema | Config | 否 | - |
+| format | String | 否 | text |
+| params | Map | 否 | - |
+| body | String | 否 | - |
+| json_field | Config | 否 | - |
+| content_field | String | 否 | - |
+| poll_interval_millis | int | 否 | - |
+| retry | int | 否 | - |
+| retry_backoff_multiplier_ms | int | 否 | 100 |
+| retry_backoff_max_ms | int | 否 | 10000 |
+| json_filed_missed_return_null | boolean | 否 | false |
+| enable_multi_lines | boolean | 否 | false |
+| common-options | config | 否 | - |
+
+`pageing` 出现在选项规则中，但本连接器会在启动时拒绝它 —— 见[分页](#分页)。
+
+### url [String]
+
+要读取的 Shopify Admin API 接口地址，例如
`https://your-store.myshopify.com/admin/api/2024-01/products.json`。
+
+### access_token [String]
+
+Shopify Admin API 的访问令牌，通过 `X-Shopify-Access-Token` 请求头发送。获取方式请参考 [Shopify
认证文档](https://shopify.dev/docs/api/admin-rest#authentication)。
+
+### method [String]
+
+http 请求方法，仅支持 GET、POST 方法。
+
+### headers [Map]
+
+额外的 HTTP 请求头。连接器已经会根据 `access_token` 设置 `X-Shopify-Access-Token`，并设置
+`Accept: application/json`，因此这里只需要配置这两者之外的请求头；在这里设置
+`X-Shopify-Access-Token` 会被 `access_token` 覆盖。
+
+### schema [Config]
+
+数据的结构，包括字段名称和字段类型。更多详情请参考 [Schema
Feature](../../introduction/concepts/schema-feature.md)。
+
+### format [String]
+
+上游数据的格式，目前仅支持 `json` 和 `text`，默认 `text`。
+
+### params [Map]
+
+http 请求参数。
+
+### json_field [Config]
+
+该参数用于配置 schema，因此必须与 schema 一起使用。它将响应中的 JSON 路径映射到 schema 字段。详情和示例请参考 [Http
source](./Http.md) 连接器。
+
+### content_field [String]
+
+该参数可以在映射为行之前，提取 JSON 响应中的某个子部分（例如顶层键 `products` 或 `orders` 下的数组）。详情和示例请参考
[Http source](./Http.md) 连接器。
+
+### common options
+
+数据源插件通用参数，详情请参考 [Source Common
Options](../common-options/source-common-options.md)。
+
+## 分页
+
+**不支持。** `pageing` 继承自 HTTP source 的选项规则，但本连接器不会把它传给 reader；若照单
+接受，作业只会读取第一次响应的数据却仍报告成功。因此配置该选项会在启动时以 `HTTP-03` 失败，
+而不是静默返回不完整的数据。
+
+仅仅把继承来的分页参数传下去也解决不了问题：共享实现是用 JsonPath 从响应*体*中读取下一个游标，
+而 Admin REST API 把游标放在 `Link` 响应头里。要真正支持，需要让 `connector-http-base`
+能够从响应头读取游标。
+
+## 示例
+
+```hocon
+source {
+ Shopify {
+ url = "https://your-store.myshopify.com/admin/api/2024-01/products.json";
+ access_token = "${SHOPIFY_ACCESS_TOKEN}"
+ method = "GET"
+ format = "json"
+ content_field = "$.products.*"
+ schema = {
+ fields {
+ id = string
+ title = string
+ vendor = string
+ product_type = string
+ created_at = string
+ updated_at = string
+ }
+ }
+ }
+}
+```
+
+`${SHOPIFY_ACCESS_TOKEN}` 是 SeaTunnel 的配置变量，不是环境变量 —— 只有在命令行传入对应的值时才会被替换：
+
+```bash
+./bin/seatunnel.sh -c your_app.conf -i SHOPIFY_ACCESS_TOKEN=shpat_xxx
+```
+
+若不传 `-i`，字面量 `${SHOPIFY_ACCESS_TOKEN}` 会被当作令牌发送，Shopify 将返回
`401`。参见[变量配置](../../introduction/concepts/config.md)。
+
+
+## 变更日志
+
+<ChangeLog />
diff --git a/plugin-mapping.properties b/plugin-mapping.properties
index e04731eb67..a340fd6fb9 100644
--- a/plugin-mapping.properties
+++ b/plugin-mapping.properties
@@ -105,6 +105,7 @@ seatunnel.sink.Tablestore = connector-tablestore
seatunnel.source.Tablestore = connector-tablestore
seatunnel.source.Lemlist = connector-http-lemlist
seatunnel.source.Klaviyo = connector-http-klaviyo
+seatunnel.source.Shopify = connector-http-shopify
seatunnel.source.Zendesk = connector-http-zendesk
seatunnel.source.Stripe = connector-http-stripe
seatunnel.sink.Slack = connector-slack
diff --git a/seatunnel-connectors-v2/connector-http/pom.xml
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/pom.xml
similarity index 55%
copy from seatunnel-connectors-v2/connector-http/pom.xml
copy to seatunnel-connectors-v2/connector-http/connector-http-shopify/pom.xml
index 7061a5d53b..5449b4a8a5 100644
--- a/seatunnel-connectors-v2/connector-http/pom.xml
+++ b/seatunnel-connectors-v2/connector-http/connector-http-shopify/pom.xml
@@ -22,30 +22,19 @@
<modelVersion>4.0.0</modelVersion>
<parent>
<groupId>org.apache.seatunnel</groupId>
- <artifactId>seatunnel-connectors-v2</artifactId>
+ <artifactId>connector-http</artifactId>
<version>${revision}</version>
</parent>
- <artifactId>connector-http</artifactId>
- <packaging>pom</packaging>
- <name>SeaTunnel : Connectors V2 : Http :</name>
- <modules>
- <module>connector-http-base</module>
- <module>connector-http-feishu</module>
- <module>connector-http-wechat</module>
- <module>connector-http-myhours</module>
- <module>connector-http-lemlist</module>
- <module>connector-http-klaviyo</module>
- <module>connector-http-onesignal</module>
- <module>connector-http-jira</module>
- <module>connector-http-gitlab</module>
- <module>connector-http-github</module>
- <module>connector-http-notion</module>
- <module>connector-http-persistiq</module>
- <module>connector-http-airtable</module>
- <module>connector-http-posthog</module>
- <module>connector-http-zendesk</module>
- <module>connector-http-stripe</module>
- </modules>
+ <artifactId>connector-http-shopify</artifactId>
+ <name>SeaTunnel : Connectors V2 : Http : Shopify</name>
+
+ <dependencies>
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>connector-http-base</artifactId>
+ <version>${project.version}</version>
+ </dependency>
+ </dependencies>
</project>
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/ShopifySource.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/ShopifySource.java
new file mode 100644
index 0000000000..3afbdb8504
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/ShopifySource.java
@@ -0,0 +1,87 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify.source;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.api.table.type.SeaTunnelRow;
+import
org.apache.seatunnel.connectors.seatunnel.common.source.AbstractSingleSplitReader;
+import
org.apache.seatunnel.connectors.seatunnel.common.source.SingleSplitReaderContext;
+import org.apache.seatunnel.connectors.seatunnel.http.config.HttpSourceOptions;
+import
org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorErrorCode;
+import
org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSource;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceReader;
+import
org.apache.seatunnel.connectors.seatunnel.shopify.source.config.ShopifySourceParameter;
+
+import lombok.extern.slf4j.Slf4j;
+
+@Slf4j
+/**
+ * Reads a Shopify Admin REST API resource — orders, products, customers —
into SeaTunnel rows.
+ *
+ * <p>A thin wrapper over {@link HttpSource}: the only behaviour it adds is
authentication, through
+ * {@link ShopifySourceParameter}, which puts the configured token in {@code
+ * X-Shopify-Access-Token}. Everything else — schema, format, retries — comes
from the HTTP source
+ * unchanged.
+ */
+public class ShopifySource extends HttpSource {
+ private final ShopifySourceParameter shopifySourceParameter = new
ShopifySourceParameter();
+
+ public ShopifySource(ReadonlyConfig pluginConfig) {
+ super(pluginConfig);
+ rejectUnsupportedPagination(pluginConfig);
+ this.shopifySourceParameter.buildWithConfig(pluginConfig);
+ }
+
+ /**
+ * {@code pageing} is inherited from the HTTP source option rule, but this
connector never hands
+ * a {@code PageInfo} to its reader, so accepting the option would read
the first response and
+ * report success — a store with more than one page of results would lose
the rest silently.
+ * Failing at startup keeps that from looking like a healthy job.
+ *
+ * <p>The Admin REST API returns its cursor in the {@code Link} response
header, while the
+ * shared reader resolves cursors with a JsonPath against the response
body, so supporting this
+ * properly needs a header-cursor mode in {@code connector-http-base}
rather than a change here.
+ */
+ private static void rejectUnsupportedPagination(ReadonlyConfig
pluginConfig) {
+ if (pluginConfig.getOptional(HttpSourceOptions.PAGEING).isPresent()) {
+ throw new HttpConnectorException(
+ HttpConnectorErrorCode.CONFIG_VALIDATION_FAILED,
+ "The Shopify source does not support the 'pageing' option:
it would be "
+ + "accepted and then ignored, so only the first
page of results would "
+ + "be read while the job reported success. Remove
'pageing' from the "
+ + "source configuration.");
+ }
+ }
+
+ @Override
+ public String getPluginName() {
+ return "Shopify";
+ }
+
+ @Override
+ public AbstractSingleSplitReader<SeaTunnelRow> createReader(
+ SingleSplitReaderContext readerContext) throws Exception {
+ return new HttpSourceReader(
+ this.shopifySourceParameter,
+ readerContext,
+ this.deserializationSchema,
+ jsonField,
+ contentField);
+ }
+}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/ShopifySourceFactory.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/ShopifySourceFactory.java
new file mode 100644
index 0000000000..acc7129216
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/ShopifySourceFactory.java
@@ -0,0 +1,64 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify.source;
+
+import org.apache.seatunnel.api.configuration.util.OptionRule;
+import org.apache.seatunnel.api.source.SeaTunnelSource;
+import org.apache.seatunnel.api.source.SourceSplit;
+import org.apache.seatunnel.api.table.connector.TableSource;
+import org.apache.seatunnel.api.table.factory.Factory;
+import org.apache.seatunnel.api.table.factory.TableSourceFactoryContext;
+import org.apache.seatunnel.connectors.seatunnel.http.source.HttpSourceFactory;
+import
org.apache.seatunnel.connectors.seatunnel.shopify.source.config.ShopifySourceOptions;
+
+import com.google.auto.service.AutoService;
+
+import java.io.Serializable;
+
+@AutoService(Factory.class)
+/**
+ * Registers {@code Shopify} as a source, discovered through {@code
@AutoService} and {@code
+ * plugin-mapping.properties}. The option rule is the HTTP source's plus a
required {@code
+ * access_token}.
+ */
+public class ShopifySourceFactory extends HttpSourceFactory {
+ @Override
+ public String factoryIdentifier() {
+ return "Shopify";
+ }
+
+ @Override
+ public <T, SplitT extends SourceSplit, StateT extends Serializable>
+ TableSource<T, SplitT, StateT>
createSource(TableSourceFactoryContext context) {
+ return () -> (SeaTunnelSource<T, SplitT, StateT>) new
ShopifySource(context.getOptions());
+ }
+
+ /**
+ * Without this the factory reports {@code HttpSource.class}, inherited
from {@link
+ * HttpSourceFactory}, so anything reading factory metadata sees the wrong
source type.
+ */
+ @Override
+ public Class<? extends SeaTunnelSource> getSourceClass() {
+ return ShopifySource.class;
+ }
+
+ @Override
+ public OptionRule optionRule() {
+ return
getHttpBuilder().required(ShopifySourceOptions.ACCESS_TOKEN).build();
+ }
+}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/config/ShopifySourceOptions.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/config/ShopifySourceOptions.java
new file mode 100644
index 0000000000..2b8b4911e2
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/config/ShopifySourceOptions.java
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify.source.config;
+
+import org.apache.seatunnel.api.configuration.Option;
+import org.apache.seatunnel.api.configuration.Options;
+import org.apache.seatunnel.connectors.seatunnel.http.config.HttpCommonOptions;
+
+/**
+ * The options this connector adds to the HTTP source's own: the Admin API
access token, and the
+ * header names used to send it.
+ */
+public class ShopifySourceOptions extends HttpCommonOptions {
+ public static final String ACCESS_TOKEN_HEADER = "X-Shopify-Access-Token";
+ public static final String ACCEPT = "Accept";
+ public static final String APPLICATION_JSON = "application/json";
+
+ public static final Option<String> ACCESS_TOKEN =
+ Options.key("access_token")
+ .stringType()
+ .noDefaultValue()
+ .withDescription("Shopify Admin API access token");
+}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/config/ShopifySourceParameter.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/config/ShopifySourceParameter.java
new file mode 100644
index 0000000000..fabe305cd0
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/main/java/org/apache/seatunnel/connectors/seatunnel/shopify/source/config/ShopifySourceParameter.java
@@ -0,0 +1,43 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify.source.config;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import org.apache.seatunnel.connectors.seatunnel.http.config.HttpParameter;
+
+import java.util.HashMap;
+
+/**
+ * Adds Shopify's authentication to the base HTTP parameters: the Admin API
expects its token in the
+ * {@code X-Shopify-Access-Token} header rather than in {@code Authorization}.
Any headers the user
+ * configured are preserved.
+ */
+public class ShopifySourceParameter extends HttpParameter {
+
+ @Override
+ public void buildWithConfig(ReadonlyConfig pluginConfig) {
+ super.buildWithConfig(pluginConfig);
+ // put the Shopify Admin API access token in headers
+ this.headers = this.getHeaders() == null ? new HashMap<>() :
this.getHeaders();
+ this.headers.put(ShopifySourceOptions.ACCEPT,
ShopifySourceOptions.APPLICATION_JSON);
+ this.headers.put(
+ ShopifySourceOptions.ACCESS_TOKEN_HEADER,
+ pluginConfig.get(ShopifySourceOptions.ACCESS_TOKEN));
+ this.setHeaders(this.headers);
+ }
+}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifyFactoryTest.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifyFactoryTest.java
new file mode 100644
index 0000000000..2c49fe4941
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifyFactoryTest.java
@@ -0,0 +1,31 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify;
+
+import
org.apache.seatunnel.connectors.seatunnel.shopify.source.ShopifySourceFactory;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+class ShopifyFactoryTest {
+
+ @Test
+ void optionRule() {
+ Assertions.assertNotNull((new ShopifySourceFactory()).optionRule());
+ }
+}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifySourceParameterTest.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifySourceParameterTest.java
new file mode 100644
index 0000000000..d65a7b6f5f
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifySourceParameterTest.java
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import
org.apache.seatunnel.connectors.seatunnel.shopify.source.config.ShopifySourceParameter;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+class ShopifySourceParameterTest {
+
+ @Test
+ void buildWithConfigSetsAccessTokenHeaderAndPreservesExistingHeaders() {
+ Map<String, String> customHeaders = new HashMap<>();
+ customHeaders.put("X-Custom-Header", "custom-value");
+
+ Map<String, Object> options = new HashMap<>();
+ options.put("url",
"https://your-store.myshopify.com/admin/api/2024-01/products.json";);
+ options.put("access_token", "shpat_example_token");
+ options.put("headers", customHeaders);
+
+ ShopifySourceParameter parameter = new ShopifySourceParameter();
+ parameter.buildWithConfig(ReadonlyConfig.fromMap(options));
+
+ Map<String, String> headers = parameter.getHeaders();
+ // the Shopify Admin API access token is sent in the
X-Shopify-Access-Token header
+ Assertions.assertEquals("shpat_example_token",
headers.get("X-Shopify-Access-Token"));
+ // requests ask for a JSON response
+ Assertions.assertEquals("application/json", headers.get("Accept"));
+ // existing custom headers are preserved
+ Assertions.assertEquals("custom-value",
headers.get("X-Custom-Header"));
+ }
+}
diff --git
a/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifySourceTest.java
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifySourceTest.java
new file mode 100644
index 0000000000..215c470853
--- /dev/null
+++
b/seatunnel-connectors-v2/connector-http/connector-http-shopify/src/test/java/org/apache/seatunnel/connectors/seatunnel/shopify/ShopifySourceTest.java
@@ -0,0 +1,66 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.seatunnel.connectors.seatunnel.shopify;
+
+import org.apache.seatunnel.api.configuration.ReadonlyConfig;
+import
org.apache.seatunnel.connectors.seatunnel.http.exception.HttpConnectorException;
+import org.apache.seatunnel.connectors.seatunnel.shopify.source.ShopifySource;
+
+import org.junit.jupiter.api.Assertions;
+import org.junit.jupiter.api.Test;
+
+import java.util.HashMap;
+import java.util.Map;
+
+class ShopifySourceTest {
+
+ private static Map<String, Object> baseOptions() {
+ Map<String, Object> options = new HashMap<>();
+ options.put("url",
"https://your-store.myshopify.com/admin/api/2024-01/products.json";);
+ options.put("access_token", "shpat_example_token");
+ return options;
+ }
+
+ @Test
+ void aConfigWithoutPagingBuilds() {
+ Assertions.assertDoesNotThrow(
+ () -> new
ShopifySource(ReadonlyConfig.fromMap(baseOptions())));
+ }
+
+ @Test
+ void pagingIsRejectedRatherThanSilentlyIgnored() {
+ // The reader never receives a PageInfo, so accepting `pageing` would
read the first
+ // response and report success — a store with more than one page would
lose the rest
+ // without anything in the job saying so. Failing here is the point of
the check.
+ Map<String, Object> paging = new HashMap<>();
+ paging.put("page_field", "page");
+ paging.put("batch_size", "100");
+
+ Map<String, Object> options = baseOptions();
+ options.put("pageing", paging);
+
+ HttpConnectorException thrown =
+ Assertions.assertThrows(
+ HttpConnectorException.class,
+ () -> new
ShopifySource(ReadonlyConfig.fromMap(options)));
+
+ Assertions.assertTrue(
+ thrown.getMessage().contains("pageing"),
+ "the message should name the option so the fix is obvious: " +
thrown.getMessage());
+ }
+}
diff --git a/seatunnel-connectors-v2/connector-http/pom.xml
b/seatunnel-connectors-v2/connector-http/pom.xml
index 7061a5d53b..d45f5c4294 100644
--- a/seatunnel-connectors-v2/connector-http/pom.xml
+++ b/seatunnel-connectors-v2/connector-http/pom.xml
@@ -44,6 +44,7 @@
<module>connector-http-persistiq</module>
<module>connector-http-airtable</module>
<module>connector-http-posthog</module>
+ <module>connector-http-shopify</module>
<module>connector-http-zendesk</module>
<module>connector-http-stripe</module>
</modules>
diff --git a/seatunnel-dist/pom.xml b/seatunnel-dist/pom.xml
index 15156d53f8..11a4681692 100644
--- a/seatunnel-dist/pom.xml
+++ b/seatunnel-dist/pom.xml
@@ -302,6 +302,12 @@
<version>${project.version}</version>
<scope>provided</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>connector-http-shopify</artifactId>
+ <version>${project.version}</version>
+ <scope>provided</scope>
+ </dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-http-zendesk</artifactId>
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml
index c68e9a14e4..4c5f122512 100644
--- a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml
+++ b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/pom.xml
@@ -105,6 +105,12 @@
<version>${project.version}</version>
<scope>test</scope>
</dependency>
+ <dependency>
+ <groupId>org.apache.seatunnel</groupId>
+ <artifactId>connector-http-shopify</artifactId>
+ <version>${project.version}</version>
+ <scope>test</scope>
+ </dependency>
<dependency>
<groupId>org.apache.seatunnel</groupId>
<artifactId>connector-http-github</artifactId>
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java
index 917844362f..5d6f44ee3a 100644
---
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/java/org/apache/seatunnel/e2e/connector/http/HttpIT.java
@@ -316,6 +316,10 @@ public class HttpIT extends TestSuiteBase implements
TestResource {
Container.ExecResult execResult11 =
container.executeJob("/persistiq_json_to_assert.conf");
Assertions.assertEquals(0, execResult11.getExitCode());
+ // http shopify
+ Container.ExecResult execResult24 =
container.executeJob("/shopify_json_to_assert.conf");
+ Assertions.assertEquals(0, execResult24.getExitCode());
+
// http httpMultiLine
Container.ExecResult execResult12 =
container.executeJob("/http_multilinejson_to_assert.conf");
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json
index b20b25c093..57c91033b4 100644
---
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/mockserver-config.json
@@ -3983,6 +3983,63 @@
}
}
},
+ {
+ "httpRequest": {
+ "method" : "GET",
+ "path": "/admin/api/2024-01/products.json",
+ "headers": {
+ "X-Shopify-Access-Token": [
+ "e2e-shopify-access-token"
+ ]
+ }
+ },
+ "httpResponse": {
+ "body": {
+ "products": [
+ {
+ "id": "7982391918849",
+ "title": "Ocean Blue Shirt",
+ "vendor": "Acme",
+ "product_type": "Shirts",
+ "created_at": "2024-01-01T10:15:30-05:00",
+ "updated_at": "2024-01-02T10:15:30-05:00"
+ },
+ {
+ "id": "7982391951617",
+ "title": "Classic Varsity Top",
+ "vendor": "Acme",
+ "product_type": "Tops",
+ "created_at": "2024-01-02T10:15:30-05:00",
+ "updated_at": "2024-01-03T10:15:30-05:00"
+ },
+ {
+ "id": "7982391984385",
+ "title": "Mist Grey Hoodie",
+ "vendor": "Northwind",
+ "product_type": "Hoodies",
+ "created_at": "2024-01-03T10:15:30-05:00",
+ "updated_at": "2024-01-04T10:15:30-05:00"
+ },
+ {
+ "id": "7982392017153",
+ "title": "Draft Denim Jacket",
+ "vendor": "Northwind",
+ "product_type": "Jackets",
+ "created_at": "2024-01-04T10:15:30-05:00",
+ "updated_at": "2024-01-05T10:15:30-05:00"
+ },
+ {
+ "id": "7982392049921",
+ "title": "Pink Ruffle Dress",
+ "vendor": "Contoso",
+ "product_type": "Dresses",
+ "created_at": "2024-01-05T10:15:30-05:00",
+ "updated_at": "2024-01-06T10:15:30-05:00"
+ }
+ ]
+ }
+ }
+ },
{
"httpRequest": {
"method" : "GET",
diff --git
a/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/shopify_json_to_assert.conf
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/shopify_json_to_assert.conf
new file mode 100644
index 0000000000..06655b1183
--- /dev/null
+++
b/seatunnel-e2e/seatunnel-connector-v2-e2e/connector-http-e2e/src/test/resources/shopify_json_to_assert.conf
@@ -0,0 +1,97 @@
+#
+# Licensed to the Apache Software Foundation (ASF) under one or more
+# contributor license agreements. See the NOTICE file distributed with
+# this work for additional information regarding copyright ownership.
+# The ASF licenses this file to You under the Apache License, Version 2.0
+# (the "License"); you may not use this file except in compliance with
+# the License. You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+env {
+ parallelism = 1
+ job.mode = "BATCH"
+}
+
+source {
+ # Mirrors the configuration documented in
docs/en/connectors/source/Shopify.md, including
+ # content_field: the Admin REST API answers with {"products": [...]} rather
than a bare
+ # array, so the JsonPath is what turns the response into rows.
+ #
+ # The mock only answers when X-Shopify-Access-Token is present, so this also
covers the one
+ # thing the connector adds over a plain HTTP source — if buildWithConfig
ever stops
+ # injecting the header, the request does not match and this job fails.
+ Shopify {
+ plugin_output = "http"
+ url = "http://mockserver:1080/admin/api/2024-01/products.json";
+ access_token = "e2e-shopify-access-token"
+ method = "GET"
+ format = "json"
+ content_field = "$.products.*"
+ schema = {
+ fields {
+ id = string
+ title = string
+ vendor = string
+ product_type = string
+ created_at = string
+ updated_at = string
+ }
+ }
+ }
+}
+
+sink {
+ Assert {
+ plugin_input = "http"
+ rules {
+ row_rules = [
+ {
+ rule_type = MAX_ROW
+ rule_value = 5
+ },
+ {
+ rule_type = MIN_ROW
+ rule_value = 5
+ }
+ ],
+
+ field_rules = [
+ {
+ field_name = id
+ field_type = string
+ field_value = [
+ {
+ rule_type = NOT_NULL
+ }
+ ]
+ },
+ {
+ field_name = title
+ field_type = string
+ field_value = [
+ {
+ rule_type = NOT_NULL
+ }
+ ]
+ },
+ {
+ field_name = product_type
+ field_type = string
+ field_value = [
+ {
+ rule_type = NOT_NULL
+ }
+ ]
+ }
+ ]
+ }
+ }
+}
