Hisoka-X commented on code in PR #8505: URL: https://github.com/apache/seatunnel/pull/8505#discussion_r1915912361
########## docs/zh/connector-v2/source/Mysql.md: ########## @@ -0,0 +1,314 @@ +# MySQL + +> JDBC Mysql 源连接器 + +## 描述 + +通过 JDBC 读取外部数据源数据。 + +## 支持 Mysql 版本 + +- 5.5/5.6/5.7/8.0/8.1/8.2/8.3/8.4 + +## 支持的引擎 + +> Spark<br/> +> Flink<br/> +> SeaTunnel Zeta<br/> + +## 需要的依赖项 + +### 对于 Spark/Flink 引擎 + +> 1. 您需要确保 [jdbc 驱动程序 jar 包](https://mvnrepository.com/artifact/mysql/mysql-connector-java) 已放置在目录 `${SEATUNNEL_HOME}/plugins/` 中。 + +### 对于 SeaTunnel Zeta 引擎 + +> 1. 您需要确保 [jdbc 驱动程序 jar 包](https://mvnrepository.com/artifact/mysql/mysql-connector-java) 已放置在目录 `${SEATUNNEL_HOME}/lib/` 中。 + +## 主要功能 + +- [x] [批处理](../../concept/connector-v2-features.md) +- [ ] [流处理](../../concept/connector-v2-features.md) +- [x] [精确一次](../../concept/connector-v2-features.md) +- [x] [列投影](../../concept/connector-v2-features.md) +- [x] [并行度](../../concept/connector-v2-features.md) +- [x] [支持用户定义的拆分](../../concept/connector-v2-features.md) +- [x] [支持多表读取](../../concept/connector-v2-features.md) + +> 支持 SQL 查询，并能实现列投影效果 + +## 支持的数据源信息 + +| 数据源 | 支持的版本 | 驱动器 | 网址 | Maven下载链接 | +|-----|---------------------------------------------------------|--------------------------|---------------------------------------|---------------------------------------------------------------------| +| Mysql | 不同的依赖版本具有不同的驱动程序类。 | com.mysql.cj.jdbc.Driver | jdbc:mysql://localhost:3306:3306/test | [下载](https://mvnrepository.com/artifact/mysql/mysql-connector-java) | + +## 数据类型映射 + +| Mysql 数据类型 | SeaTunnel 数据类型 | +|---------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------| +| BIT(1)<br/>TINYINT(1) | BOOLEAN | +| TINYINT | BYTE | +| TINYINT UNSIGNED<br/>SMALLINT | SMALLINT | +| SMALLINT UNSIGNED<br/>MEDIUMINT<br/>MEDIUMINT UNSIGNED<br/>INT<br/>INTEGER<br/>YEAR | INT | +| INT UNSIGNED<br/>INTEGER UNSIGNED<br/>BIGINT | BIGINT | +| BIGINT UNSIGNED | DECIMAL(20,0) | +| DECIMAL(x,y)(获取指定列的列大小<38) | DECIMAL(x,y) | +| DECIMAL(x,y)(获取指定列的列大小>38) | DECIMAL(38,18) | +| DECIMAL UNSIGNED | DECIMAL((获取指定列的列大小)+1,<br/>(获取指定列的小数点右侧的位数)) | +| FLOAT<br/>FLOAT UNSIGNED | FLOAT | +| DOUBLE<br/>DOUBLE UNSIGNED | DOUBLE | +| CHAR<br/>VARCHAR<br/>TINYTEXT<br/>MEDIUMTEXT<br/>TEXT<br/>LONGTEXT<br/>JSON<br/>ENUM | STRING | +| DATE | DATE | +| TIME(s) | TIME(s) | +| DATETIME<br/>TIMESTAMP(s) | TIMESTAMP(s) | +| TINYBLOB<br/>MEDIUMBLOB<br/>BLOB<br/>LONGBLOB<br/>BINARY<br/>VARBINAR<br/>BIT(n)<br/>GEOMETRY | BYTES | + +## 数据源参数 + +| 名称 | 类型 | 是否必填 | 默认值 | 描述 | +|--------------------------------------------|------------|------|-----------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------| +| url | String | 是 | - | JDBC 连接的 URL。参见一个示例: <br>`jdbc:mysql://localhost:3306:3306/test`。 | Review Comment: ```suggestion | url | String | 是 | - | JDBC 连接的 URL。参见示例: <br>`jdbc:mysql://localhost:3306:3306/test`。 | ``` ########## docs/zh/other-engine/spark.md: ########## @@ -0,0 +1,107 @@ +# SeaTunnel 通过 Spark 引擎运行 + +Spark 是一个强大的高性能分布式计算处理引擎。有关它的更多信息，您可以搜索"Apache Spark" + + +### 如何在作业中设置 Spark 配置信息 + +例： +我为某项工作设置了一些 spark 配置项 Review Comment: ```suggestion 我为这个任务设置了一些 spark 配置项 ``` ########## docs/zh/other-engine/spark.md: ########## @@ -0,0 +1,107 @@ +# SeaTunnel 通过 Spark 引擎运行 + +Spark 是一个强大的高性能分布式计算处理引擎。有关它的更多信息，您可以搜索"Apache Spark" + + +### 如何在作业中设置 Spark 配置信息 + +例： +我为某项工作设置了一些 spark 配置项 + +``` +env { + spark.app.name = "example" + spark.sql.catalogImplementation = "hive" + spark.executor.memory= "2g" + spark.executor.instances = "2" + spark.yarn.priority = "100' + hive.exec.dynamic.partition.mode = "nonstrict" + spark.dynamicAllocation.enabled="false" +} +``` + +### 命令行示例 + +#### Spark on Yarn集群 + +``` +./bin/start-seatunnel-spark-3-connector-v2.sh --master yarn --deploy-mode cluster --config config/example.conf +``` + +#### Spark on Yarn集群 + +``` +./bin/start-seatunnel-spark-3-connector-v2.sh --master yarn --deploy-mode client --config config/example.conf +``` + +### 如何设置简单的 Spark 作业 + +这是通过 Spark 运行的一个简单作业。会将随机生成的数据输出到控制台 + +``` +env { + # common parameter + parallelism = 1 + + # spark special parameter + spark.app.name = "example" + spark.sql.catalogImplementation = "hive" + spark.executor.memory= "2g" + spark.executor.instances = "1" + spark.yarn.priority = "100" + hive.exec.dynamic.partition.mode = "nonstrict" + spark.dynamicAllocation.enabled="false" +} + +source { + FakeSource { + schema = { + fields { + c_map = "map<string, array<int>>" + c_array = "array<int>" + c_string = string + c_boolean = boolean + c_tinyint = tinyint + c_smallint = smallint + c_int = int + c_bigint = bigint + c_float = float + c_double = double + c_decimal = "decimal(30, 8)" + c_null = "null" + c_bytes = bytes + c_date = date + c_timestamp = timestamp + c_row = { + c_map = "map<string, map<string, string>>" + c_array = "array<int>" + c_string = string + c_boolean = boolean + c_tinyint = tinyint + c_smallint = smallint + c_int = int + c_bigint = bigint + c_float = float + c_double = double + c_decimal = "decimal(30, 8)" + c_null = "null" + c_bytes = bytes + c_date = date + c_timestamp = timestamp + } + } + } +} +} + +transform { +} + +sink{ + Console{} +} +``` + +### 如何在项目中运行作业 + +将代码拉取到本地后，进入 seatunnel-examples/seatunnel-spark-connector-v2-example 模块，找到 org.apache.seatunnel.example.spark.v2.SeaTunnelApiExample 来完成作业的操作。 Review Comment: ```suggestion 将代码拉取到本地后，进入 seatunnel-examples/seatunnel-spark-connector-v2-example 模块，找到 org.apache.seatunnel.example.spark.v2.SeaTunnelApiExample 来完成作业的运行。 ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
