This is an automated email from the ASF dual-hosted git repository.
lidongdai pushed a change to branch docs-architecture-design
in repository https://gitbox.apache.org/repos/asf/seatunnel.git
discard f3368c7244 docs: align EN architecture docs with Zeta implementation
discard 7cfab2e2b3 Docs: review and refine architecture documents for accuracy
discard 20ec27c880 docs(zh): align Jdbc connector name in examples
discard 378e3be9b2 docs(zh): fix schema evolution config examples
discard ca3e8c8e52 docs(zh): remove source references and java blocks
discard 76e8354ec1 Remove tasks/architecture-docs-design.md from PR
discard 44bd702c17 [Doc] Add comprehensive architecture design documentation
add 4c110c8fdf [Feature][Connector-V2][HdfsFile] Support true large-file
split for parallel read (#10332)
add 94d6f5c675 [Fix][docs]fix schema-feature.md doc error (#10413)
add 6948d3492e [Feat][Connector-v2][Kafka] Add support for Schema Registry
wire format in Protobuf deserialization (#10183)
add 70c7040746 [Improve][Common] Improve SinkFlowTestUtils Checkpoint
Logic (#10071)
add 1cfffc7094 [Fix][Connector-V2] Fix NPE when heartbeat.action.query is
configured (#10278)
add 0a4e6ae38e [Fix] [Zeta] Fix Write Count Display Issue in Multi-Sink
Scenarios (#10376)
add 9aea5c9546 [Improve][Zeta] engine-server and seantunnel-ui support
remote paginated queries (#9951)
add 07c0bd1ef3 [Fix][Zeta] Fix memory leak when cancelling pending job
(#10315)
add eca4e2b342 [Improve][Core][Metrics] Refactor and enhance job metrics
handling for multiple sinks with updated tests (#10273)
add 8e27bffff8 [Fix][Connector-v2][MongoDB] There is a problem with using
Cache for multi-task submission (#10116)
add 2be8d7bd0b docs: align EN architecture docs with Zeta implementation
This update added new revisions after undoing existing revisions.
That is to say, some revisions that were in the old version of the
branch are not in the new version. This situation occurs
when a user --force pushes a change and generates a repository
containing something like this:
* -- * -- B -- O -- O -- O (f3368c7244)
\
N -- N -- N refs/heads/docs-architecture-design (2be8d7bd0b)
You should already have received notification emails for all of the O
revisions, and so the following emails describe only the N revisions
from the common base, B.
Any revisions marked "omit" are not gone; other references still
refer to them. Any revisions marked "discard" are gone forever.
No new revisions were added by this update.
Summary of changes:
.github/workflows/backend.yml | 4 +-
docs/en/connectors/source/HdfsFile.md | 26 +
docs/en/connectors/source/Kafka.md | 56 +-
docs/en/connectors/source/LocalFile.md | 15 +-
docs/en/engines/zeta/rest-api-v2.md | 15 +-
.../introduction/concepts/incompatible-changes.md | 24 +
docs/en/introduction/concepts/schema-feature.md | 2 +-
docs/sidebars.js | 48 --
docs/zh/architecture/api-design/catalog-table.md | 393 --------------
.../architecture/api-design/sink-architecture.md | 376 --------------
.../architecture/api-design/source-architecture.md | 438 ----------------
docs/zh/architecture/data-flow/multi-table.md | 434 ----------------
docs/zh/architecture/design-philosophy.md | 449 ----------------
docs/zh/architecture/engine/engine-architecture.md | 574 ---------------------
docs/zh/architecture/engine/resource-management.md | 539 -------------------
.../fault-tolerance/checkpoint-mechanism.md | 549 --------------------
.../architecture/fault-tolerance/exactly-once.md | 504 ------------------
docs/zh/architecture/overview.md | 439 ----------------
.../architecture/translation/translation-layer.md | 251 ---------
docs/zh/connectors/source/HdfsFile.md | 26 +
docs/zh/connectors/source/Kafka.md | 54 ++
docs/zh/connectors/source/LocalFile.md | 13 +
docs/zh/engines/zeta/rest-api-v2.md | 15 +-
.../introduction/concepts/incompatible-changes.md | 24 +
docs/zh/introduction/concepts/schema-feature.md | 2 +-
.../DefaultHeartbeatConnectionProvider.java | 20 +-
.../base/relational/JdbcSourceEventDispatcher.java | 3 +
.../seatunnel/cdc/mongodb/utils/MongodbUtils.java | 17 +-
.../reader/fetch/MySqlSourceFetchTaskContext.java | 8 +
.../reader/fetch/OracleSourceFetchTaskContext.java | 11 +-
.../reader/PostgresSourceFetchTaskContext.java | 14 +
.../fetch/SqlServerSourceFetchTaskContext.java | 15 +-
.../seatunnel/sink/SinkFlowTestUtils.java | 338 +++++++++++-
.../file/exception/FileConnectorErrorCode.java | 5 +-
.../file/source/BaseMultipleTableFileSource.java | 24 +-
.../file/source/reader/AbstractReadStrategy.java | 22 +-
.../split/AccordingToSplitSizeSplitStrategy.java | 242 ++++++---
.../file/source/split/FileSourceSplit.java | 14 +
.../source/split/FileSplitStrategyFactory.java} | 23 +-
.../MultipleTableFileSourceSplitEnumerator.java | 68 ++-
.../split/MultipleTableFileSplitStrategy.java | 72 +++
.../source/split/ParquetFileSplitStrategy.java | 74 ++-
.../source/reader/AbstractReadStrategyTest.java | 81 +++
.../split/FileSourceSplitCompatibilityTest.java | 209 ++++++++
.../seatunnel/file/hdfs/source/HdfsFileSource.java | 6 +-
.../file/hdfs/source/HdfsFileSourceFactory.java | 12 +
...sFileAccordingToSplitSizeSplitStrategyTest.java | 229 ++++++++
.../split/HdfsFileSplitStrategyFactoryTest.java | 123 +++++
.../file/local/source/LocalFileSource.java | 9 +-
...LocalFileAccordingToSplitSizeSplitStrategy.java | 47 +-
.../seatunnel/file/local/LocalFileSourceTest.java | 61 ++-
.../file/local/SplitFileStrategyTest.java | 141 ++---
.../seatunnel/kafka/config/KafkaSourceOptions.java | 9 +
.../seatunnel/kafka/source/KafkaSourceConfig.java | 10 +-
.../seatunnel/kafka/source/KafkaSourceFactory.java | 5 +
.../src/test/java/mongodb/MongodbCDCIT.java | 48 ++
..._mysql.conf => mongodbcdc_to_mysql_orders.conf} | 11 +-
.../cdc/mysql/AbstractMysqlCDCITBase.java | 60 +++
....conf => mysqlcdc_to_mysql_with_heartbeat.conf} | 7 +-
.../seatunnel/cdc/oracle/OracleCDCIT.java | 34 ++
...onf => oraclecdc_to_oracle_with_heartbeat.conf} | 5 +-
.../seatunnel/cdc/postgres/PostgresCDCIT.java | 60 +++
...=> postgrescdc_to_postgres_with_heartbeat.conf} | 4 +
.../connector/cdc/sqlserver/SqlServerCDCIT.java | 163 ++++--
...=> sqlservercdc_to_console_with_heartbeat.conf} | 4 +
.../e2e/connector/file/hdfs/HdfsFileIT.java | 41 ++
...sert.conf => hdfs_parquet_split_to_assert.conf} | 3 +-
..._assert.conf => hdfs_text_split_to_assert.conf} | 16 +-
.../seatunnel/e2e/connector/kafka/KafkaIT.java | 100 ++++
...chema_registry_header_transform_to_assert.conf} | 10 +-
.../connector-mongodb-e2e/pom.xml | 7 +
.../e2e/connector/v2/mongodb/MongodbIT.java | 103 +++-
.../fake_source_to_transaction_sink_mongodb.conf | 74 ---
.../seatunnel/engine/e2e/CommittedMetricsIT.java | 17 +-
.../seatunnel/engine/e2e/MultiTableMetricsIT.java | 32 +-
.../org/apache/seatunnel/engine/e2e/RestApiIT.java | 8 +-
.../seatunnel/engine/client/job/JobClient.java | 25 +-
.../seatunnel/engine/client/JobClientTest.java | 19 +-
.../engine/core/parse/JobConfigParser.java | 6 +-
.../engine/server/CoordinatorService.java | 7 -
.../engine/server/dag/physical/PhysicalPlan.java | 1 +
.../engine/server/rest/service/BaseService.java | 332 +++++++++++-
.../engine/server/rest/service/JobInfoService.java | 4 +
.../server/rest/servlet/FinishedJobsServlet.java | 4 +-
.../server/rest/servlet/PageBaseServlet.java | 75 +++
.../server/rest/servlet/RunningJobsServlet.java | 4 +-
...CoordinatorServiceWithCancelPendingJobTest.java | 20 +-
.../engine/server/rest/RestApiHttpsTest.java | 273 +++++++++-
.../server/rest/RestApiRequestCallback.java} | 9 +-
.../rest/service/BaseServiceTableMetricsTest.java | 288 ++++++++++-
.../seatunnel-engine-ui/src/service/job/index.ts | 6 +-
.../seatunnel-engine-ui/src/service/job/types.ts | 5 +
.../seatunnel-engine-ui/src/tests/jobs.spec.ts | 26 +-
.../src/views/jobs/finished-jobs.tsx | 25 +-
.../src/views/jobs/running-jobs.tsx | 25 +-
.../protobuf/ProtobufDeserializationSchema.java | 15 +-
...RegistryAwareProtobufDeserializationSchema.java | 129 +++++
...stryAwareProtobufDeserializationSchemaTest.java | 192 +++++++
tasks/docs-zh-architecture-accuracy-review.md | 187 -------
99 files changed, 3910 insertions(+), 5756 deletions(-)
delete mode 100644 docs/zh/architecture/api-design/catalog-table.md
delete mode 100644 docs/zh/architecture/api-design/sink-architecture.md
delete mode 100644 docs/zh/architecture/api-design/source-architecture.md
delete mode 100644 docs/zh/architecture/data-flow/multi-table.md
delete mode 100644 docs/zh/architecture/design-philosophy.md
delete mode 100644 docs/zh/architecture/engine/engine-architecture.md
delete mode 100644 docs/zh/architecture/engine/resource-management.md
delete mode 100644 docs/zh/architecture/fault-tolerance/checkpoint-mechanism.md
delete mode 100644 docs/zh/architecture/fault-tolerance/exactly-once.md
delete mode 100644 docs/zh/architecture/overview.md
delete mode 100644 docs/zh/architecture/translation/translation-layer.md
copy
seatunnel-api/src/main/java/org/apache/seatunnel/api/table/catalog/SQLPreviewResult.java
=>
seatunnel-connectors-v2/connector-cdc/connector-cdc-base/src/main/java/io/debezium/heartbeat/DefaultHeartbeatConnectionProvider.java
(68%)
rename
seatunnel-connectors-v2/connector-file/{connector-file-local/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/local/source/split/LocalFileSplitStrategyFactory.java
=>
connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSplitStrategyFactory.java}
(79%)
create mode 100644
seatunnel-connectors-v2/connector-file/connector-file-base/src/main/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/MultipleTableFileSplitStrategy.java
create mode 100644
seatunnel-connectors-v2/connector-file/connector-file-base/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/source/split/FileSourceSplitCompatibilityTest.java
create mode 100644
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileAccordingToSplitSizeSplitStrategyTest.java
create mode 100644
seatunnel-connectors-v2/connector-file/connector-file-hadoop/src/test/java/org/apache/seatunnel/connectors/seatunnel/file/hdfs/source/split/HdfsFileSplitStrategyFactoryTest.java
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mongodb-e2e/src/test/resources/{mongodbcdc_to_mysql.conf
=> mongodbcdc_to_mysql_orders.conf} (89%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-mysql-e2e/src/test/resources/{mysqlcdc_to_mysql.conf
=> mysqlcdc_to_mysql_with_heartbeat.conf} (93%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-oracle-e2e/src/test/resources/{oraclecdc_to_oracle_use_select_count.conf
=> oraclecdc_to_oracle_with_heartbeat.conf} (93%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-postgres-e2e/src/test/resources/{postgrescdc_to_postgres.conf
=> postgrescdc_to_postgres_with_heartbeat.conf} (92%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-cdc-sqlserver-e2e/src/test/resources/{sqlservercdc_to_console.conf
=> sqlservercdc_to_console_with_heartbeat.conf} (91%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-hadoop-e2e/src/test/resources/{hdfs_normal_to_assert.conf
=> hdfs_parquet_split_to_assert.conf} (97%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-file-hadoop-e2e/src/test/resources/{hdfs_empty_text_to_assert.conf
=> hdfs_text_split_to_assert.conf} (73%)
copy
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-kafka-e2e/src/test/resources/protobuf/{kafka_protobuf_transform_to_assert.conf
=> kafka_protobuf_schema_registry_header_transform_to_assert.conf} (94%)
delete mode 100644
seatunnel-e2e/seatunnel-connector-v2-e2e/connector-mongodb-e2e/src/test/resources/transactionIT/fake_source_to_transaction_sink_mongodb.conf
create mode 100644
seatunnel-engine/seatunnel-engine-server/src/main/java/org/apache/seatunnel/engine/server/rest/servlet/PageBaseServlet.java
copy
seatunnel-engine/seatunnel-engine-server/src/{main/java/org/apache/seatunnel/engine/server/rest/ErrResponse.java
=>
test/java/org/apache/seatunnel/engine/server/rest/RestApiRequestCallback.java}
(87%)
create mode 100644
seatunnel-formats/seatunnel-format-protobuf/src/main/java/org/apache/seatunnel/format/protobuf/SchemaRegistryAwareProtobufDeserializationSchema.java
create mode 100644
seatunnel-formats/seatunnel-format-protobuf/src/test/java/org/apache/seatunnel/format/protobuf/SchemaRegistryAwareProtobufDeserializationSchemaTest.java
delete mode 100644 tasks/docs-zh-architecture-accuracy-review.md
