This is an automated email from the ASF dual-hosted git repository.
tyrantlucifer pushed a commit to branch dev
in repository https://gitbox.apache.org/repos/asf/incubator-seatunnel.git
The following commit(s) were added to refs/heads/dev by this push:
new eee84f7b8 [Docs][Connector-V2][TableFactory] Add TableSourceFactory
and TableSinkFactory Doc (#3343)
eee84f7b8 is described below
commit eee84f7b8e10050ddfcf9bdca68f10a7c655616f
Author: Hisoka <[email protected]>
AuthorDate: Fri Nov 11 13:54:16 2022 +0800
[Docs][Connector-V2][TableFactory] Add TableSourceFactory and
TableSinkFactory Doc (#3343)
* [Connector-V2] [Doc] Add TableSourceFactory and TableSinkFactory Doc
* [Connector-V2] [Doc] Add TableSourceFactory and TableSinkFactory Doc
* Update seatunnel-connectors-v2/README.md
Co-authored-by: Zongwen Li <[email protected]>
* Apply suggestions from code review
Co-authored-by: Eric <[email protected]>
Co-authored-by: Zongwen Li <[email protected]>
Co-authored-by: Eric <[email protected]>
---
seatunnel-connectors-v2/README.md | 35 +++++++++++++++++++++++++++++++++++
seatunnel-connectors-v2/README.zh.md | 22 ++++++++++++++++++++++
2 files changed, 57 insertions(+)
diff --git a/seatunnel-connectors-v2/README.md
b/seatunnel-connectors-v2/README.md
index 61aad40f8..dfa7badf6 100644
--- a/seatunnel-connectors-v2/README.md
+++ b/seatunnel-connectors-v2/README.md
@@ -199,6 +199,41 @@ In the current version, it is recommended to implement
``SinkAggregatedCommitter
provide strong consistency guarantee in Flink/Spark. At the same time, commit
should be idempotent, and save engine
retry can work normally.
+### TableSourceFactory and TableSinkFactory
+
+In order to automatically create the Source Connector and Sink Connector and
Transform Connector, we need the connector to return the parameters needed to
create them and the verification rules for each parameter. For Source Connector
and Sink Connector, we define `TableSourceFactory` and `TableSinkFactory`
+supported by the current connector and the required parameters. We define
TableSourceFactory and TableSinkFactory,
+It is recommended to put it in the same directory as the implementation class
of SeaTunnelSource or SeaTunnelSink for easy searching.
+
+- `factoryIdentifier` is used to indicate the name of the current Factory.
This value should be the same as the
+ value returned by `getPluginName`, so that if Factory is used to create
Source/Sink in the future,
+ A seamless switch can be achieved.
+- `createSink` and `createSource` are the methods for creating Source and Sink
respectively,
+ and do not need to be implemented at present.
+- `optionRule` returns the parameter logic, which is used to indicate which
parameters of our connector are supported,
+ which parameters are required, which parameters are optional, and which
parameters are exclusive, which parameters are bundledRequired.
+ This method will be used when we visually create the connector logic, and
it will also be used to generate a complete parameter
+ object according to the parameters configured by the user, and then the
connector developer does not need to judge whether the parameters
+ exist one by one in the Config, and use it directly That's it.
+ You can refer to existing implementations, such as
`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`.
+ There is support for configuring Schema for many Sources, so a common
Option is used.
+ If you need a schema, you can refer to
`org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema.SCHEMA`.
+
+Don't forget to add `@AutoService(Factory.class)` to the class. This Factory
is the parent class of TableSourceFactory and TableSinkFactory.
+
+### **Options**
+
+When we implement TableSourceFactory and TableSinkFactory, the corresponding
Option will be created.
+Each Option corresponds to a configuration, but different configurations will
have different types.
+Common types can be created by directly calling the corresponding method.
+But if our parameter type is an object, we can use POJO to represent
parameters of object type,
+and need to use `org.apache.seatunnel.api.configuration.util.OptionMark` on
each parameter to indicate that this is A child Option.
+`OptionMark` has two parameters, `name` is used to declare the parameter name
corresponding to the field.
+If it is empty, we will convert the small camel case corresponding to java to
underscore by default, such as: `myUserPassword` -> `my_user_password` .
+In most cases, the default is empty. `description` is used to indicate the
description of the current parameter.
+This parameter is optional. It is recommended to be consistent with the
documentation. For specific examples,
+please refer to
`org.apache.seatunnel.connectors.seatunnel.assertion.sink.AssertSinkFactory`.
+
## **Result**
All Connector implementations should be under the ``seatunnel-connectors-v2``,
and the examples that can be referred to
diff --git a/seatunnel-connectors-v2/README.zh.md
b/seatunnel-connectors-v2/README.zh.md
index 4efbc6d59..81af8be2e 100644
--- a/seatunnel-connectors-v2/README.zh.md
+++ b/seatunnel-connectors-v2/README.zh.md
@@ -149,6 +149,28 @@ Sink可以根据组件属性进行选择，到底是只实现`SinkCommitter`或`
当前版本推荐将实现SinkAggregatedCommitter作为首选，可以在Flink/Spark中提供较强的一致性保证，同时commit应该要实现幂等性，保存引擎重试能够正常运作。
+### TableSourceFactory 和 TableSinkFactory
+
+为了实现自动化的创建Source或者Sink，我们需要连接器能够声明并返回创建他们所需要的参数列表和每个参数的校验规则。为了实现这个目标，我们定义了TableSourceFactory和TableSinkFactory，
+建议将其放在和SeaTunnelSource或SeaTunnelSink实现类同一目录下，方便寻找。
+
+- `factoryIdentifier`
用于表明当前Factory的名称，这个值应该和`getPluginName`返回的值一致，这样后续如果使用Factory来创建Source/Sink，
+就能实现无缝切换。
+- `createSink` 和 `createSource` 分别是创建Source和Sink的方法，目前不用实现。
+- `optionRule`
返回的是参数逻辑，用于表示我们的连接器参数哪些支持，哪些参数是必须(required)的，哪些参数是可选(optional)的，哪些参数是互斥(exclusive)的，哪些参数是绑定(bundledRequired)的。
+这个方法会在我们可视化创建连接器逻辑的时候用到，同时也会用于根据用户配置的参数生成完整的参数对象，然后连接器开发者就不用在Config里面一个个判断参数是否存在，直接使用即可。
+可以参考现有的实现，比如`org.apache.seatunnel.connectors.seatunnel.elasticsearch.source.ElasticsearchSourceFactory`。针对很多Source都有支持配置Schema，所以采用了通用的Option，
+需要Schema则可以引用`org.apache.seatunnel.connectors.seatunnel.common.schema.SeaTunnelSchema.SCHEMA`。
+
+别忘记添加`@AutoService(Factory.class)` 到类上面。这个Factory即TableSourceFactory 和
TableSinkFactory的父类。
+
+### Option
+
+当我们实现TableSourceFactory 和
TableSinkFactory时，会创建对应的Option，每一个Option对应的就是一个配置，但是不同的配置会有不同的类型，普通类型直接调用对应的方法即可创建。
+但是如果我们参数类型是一个对象，我们就可以使用POJO来表示对象类型的参数，同时需要在每个参数上使用`org.apache.seatunnel.api.configuration.util.OptionMark`来表明这是一个子Option。
+`OptionMark`有两个参数，`name`用于声明字段对应的参数名称，如果为空的话，我们会默认将java对应的小驼峰转换成下划线进行表达，如：`myUserPassword`->`my_user_password`。
+在大多数情况下，默认为空即可。`description`用于表示当前参数的描述，这个参数是可选的，建议和文档上的保持一致。具体例子可以参考`org.apache.seatunnel.connectors.seatunnel.assertion.sink.AssertSinkFactory`。
+
## 实现
现阶段所有的连接器实现及可参考的示例都在seatunnel-connectors-v2下，用户可自行查阅参考。
