This is an automated email from the ASF dual-hosted git repository. ic4y pushed a commit to branch 2.3.3-release in repository https://gitbox.apache.org/repos/asf/seatunnel.git
commit d507378a79424e96391e6f7fa689f35fa7d709fa Author: liuli <[email protected]> AuthorDate: Tue Aug 15 22:50:31 2023 +0800 [Release][2.3.3][release-note] Add release-note --- release-note.md | 336 ++++++++++++++++++++++++++++---------------------------- 1 file changed, 165 insertions(+), 171 deletions(-) diff --git a/release-note.md b/release-note.md index d647bdbad9..c1999101f1 100644 --- a/release-note.md +++ b/release-note.md @@ -1,194 +1,188 @@ -# next-release +# 2.3.3 Release Note ## Bug fix ### Core -- [Core] [API] Fixed generic class loss for lists (#4421) -- [Core] [API] Fix parse nested row data type key changed upper (#4459) -- [Starter][Flink]Support transform-v2 for flink #3396 -- [Flink] Support flink 1.14.x #3963 -- [Core][Translation][Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type (#5170) - -### Transformer -- [Spark] Support transform-v2 for spark (#3409) -- [ALL]Add FieldMapper Transform #3781 -### Connectors -- [Elasticsearch] Support https protocol & compatible with opensearch -- [Hbase] Add hbase sink connector #4049 -### Formats -- [Canal]Support read canal format message #3950 -- [Debezium]Support debezium canal format message #3981 - -### Connector-V2 - -- [Json-format] [Canal-Json] Fix json deserialize NPE (#4195) -- [Connector-V2] [Jdbc] Field aliases are not supported in the query of jdbc source. (#4210) -- [Connector-V2] [Jdbc] Fix connection failure caused by connection timeout. (#4322) -- [Connector-V2] [Jdbc] Set default value to false of JdbcOption: generate_sink_sql (#4471) -- [Connector-V2] [JDBC] Fix TiDBCatalog without open (#4718) -- [Connector-V2] [Jdbc] Fix XA DataSource crash(Oracle/Dameng/SqlServer) (#4866) -- [Connector-V2] [Pulsar] Fix the bug that can't consume messages all the time. (#4125) -- [Connector-V2] [Eleasticsearch] Document description error (#4390) -- [Connector-V2] [Eleasticsearch] Source deserializer error and inappropriate (#4233) -- [Connector-V2] [Kafka] Fix KafkaProducer resources have never been released. (#4302) -- [Connector-V2] [Kafka] Fix the permission problem caused by client.id. (#4246) -- [Connector-V2] [Kafka] Fix KafkaConsumerThread exit caused by commit offset error. (#4379) -- [Connector-V2] [Mongodb] Mongodb support cdc sink. (#4833) -- [Connector-V2] [kafka] Fix the problem that the partition information can not be obtained when kafka is restored (#4764) -- [Connector-V2] [SFTP] Fix incorrect exception handling logic (#4720) -- [Connector-V2] [File] Fix read temp file (#4876) -- [Connector-V2] [CDC Base] Solving the ConcurrentModificationException caused by snapshotState being modified concurrently. (#4877) -- [Connector-V2] [Doris] update last checkpoint id when doing snapshot (#4881) -- [Connector-v2] [kafka] Fix the short interval of pull data settings and revise the format (#4875) -- [Connector-v2] [RabbitMQ] Fix reduplicate ack msg bug and code style (#4842) -- [Connector-V2] [Jdbc] Fix the error of extracting primary key column in sink (#4815) -- [Connector-V2] [Jdbc] Fix reconnect throw close statement exception (#4801) -- [Connector-V2] [Jdbc] Fix sqlserver system table case sensitivity (#4806) -- [Connector-v2] [File] Fix configuration file format and error comments (#4762) -- [Connector-v2] [Jdbc] Fix oracle sql table identifier (#4754) -- [Connector-v2] [Clickhouse] fix get clickhouse local table name with closing bracket from distributed table engineFull (#4710) -- [Connector-v2] [CDC] Fix jdbc connection leak for mysql (#5037) +- [Translation] [Spark] Fix SeaTunnelRowConvertor fail to convert when schema contains row type (#5170) + +### Connector +- [Connector] [Jdbc] Fix the problem of JdbcOutputFormat database connection leak (#4802) +- [Connector-V2] [AmazonDynamoDB] Fix the problem that all table data cannot be obtained (#5146) +- [Connector-V2] [CDC-base] Fix cdc connection does not close (#4922) +- [Connector-V2] [Clickhouse] Fix clickhouse source connector read Nullable() type is not null,example:Nullable(Float64) while value is null the result is 0.0 (#5080) +- [Connector-V2] [ClickhouseFile] Fix ClickhouseFile write file failed when field value is null (#4937) +- [Connector-V2] [Doris] Add streamload Content-type for doris URLdecode error (#4880) +- [Connector-V2] [E2E] Fix maven scope (#4901) +- [Connector-V2] [File-Base] Fix Hadoop Kerberos authentication related issues (#5171) +- [Connector-V2] [Flink-runtime] Flink register table Environment The running mode is set to`job.mode` (#4826) +- [Connector-V2] [HbaseSink] Fix default timestamp (#4958) +- [Connector-V2] [Hive] Support user-defined hive-site.xml (#4965) +- [Connector-V2] [Jdbc] Support postgresql json type (#5194) +- [Connector-V2] [Jdbc] Fix XA Transaction bug (#5020) +- [Connector-V2] [Jdbc] Modify error comment in JdbcSourceConfigFactory (#5088) +- [Connector-V2] [Jdbc-Oracle] Fix the Oracle number type mapping problem (#5209) +- [Connector-V2] [Jdbc-Postgres] PostgreSQL versions below 9.5 are compatible use cdc sync problem (#5120) +- [Connector-V2] [JindoOssFile] Fix plugin-mapping.properties (#5215) +- [Connector-V2] [LocalFileSink] Fix LocalFile Sink file_format_type (#5118) +- [Connector-V2] [Mongodb] Fix document error content and remove redundant code (#4982) +- [Connector-V2] [Mongodb] Compatible with historical parameters (#4997) +- [Connector-V2] [MongoDB-CDC] Incremental data kind error in snapshot phase (#5184) +- [Connector-V2] [MongoDB-CDC] Solve startup resume token is negative (#5143) +- [Connector-V2] [MongoDB-CDC] Refine data format to adapt to universal logic (#5162) +- [Connector-V2] [MongoDB-CDC] Fix array index anomalies caused by #5057 (#5195) +- [Connector-V2] [MySql-CDC] Fix jdbc connection leak for mysql (#5037) +- [Connector-V2] [MySql-CDC] Fix mysql bit column to java byte (#4817) +- [Connector-V2] [MySql-CDC] Fix listener not released when BinlogClient reuse (#5011) +- [Connector-V2] [MySql-CDC] ServerId from int to long (#5033) (#5035) +- [Connector-V2] [Paimon] Bump paimon-bundle version to 0.4.0-incubating (#5219) +- [Connector-V2] [Redis] Fix redis nodes format error (#4981) +- [Connector-V2] [StarRocks] Fix StarRocksJsonSerializer will transform array/map/row to string (#5281) +- [Connector-V2] [StarRocks] Fix code style (#4966) ### Zeta(ST-Engine) - -- [Zeta] Fix LogicalDagGeneratorTest testcase (#4401) -- [Zeta] Fix MultipleTableJobConfigParser parse only one transform (#4412) -- [Zeta] Fix missing common plugin jars (#4448) -- [Zeta] Fix handleCheckpointError be called while checkpoint already complete (#4442) -- [Zeta] Fix job error message is not right bug (#4463) -- [Zeta] Fix finding TaskGroup deployment node bug (#4449) -- [Zeta] Fix the bug of conf (#4488) -- [Zeta] Fix Connector load logic from zeta (#4510) -- [Zeta] Fix conflict dependency of hadoop-hdfs (#4509) -- [Zeta] Fix TaskExecutionService synchronized lock will not release (#4886) -- [Zeta] Fix TaskExecutionService will return not active ExecutionContext (#4869) -- [Zeta] Fix deploy operation timeout but task already finished bug (#4867) -- [Zeta] Fix restoreComplete Future can't be completed when cancel task (#4863) -- [Zeta] Fix IMap operation timeout bug (#4859) -- [Zeta] fix pipeline state not right bug (#4823) -- [Zeta] Fix the incorrect setting of transform parallelism (#4814) -- [Zeta] Fix master active bug (#4855) -- [Zeta] Fix completePendingCheckpoint concurrent action (#4854) -- [Zeta] Fix engine runtime error (#4850) -- [Zeta] Fix TaskGroupContext always hold classloader so classloader can't recycle (#4849) -- [Zeta] Fix task `notifyTaskStatusToMaster` failed when job not running or failed before run (#4847) -- [Zeta] Fix cpu load problem (#4828) -- [zeta] Fix the deadlock issue with JDBC driver loading (#4878) -- [zeta] dynamically replace the value of the variable at runtime (#4950) - -### E2E - -- [E2E] [Kafka] Fix kafka e2e testcase (#4520) -- [Container Version] Fix risk of unreproducible test cases #4591 -- [E2e] [Mysql-cdc] Removing the excess MySqlIncrementalSourceIT e2e reduces the CI time (#4738) +- [Zeta] Resolved the issue causing checkpoints to halt on tolerable-failure=0 (#5263) +- [Zeta] Fix the issue of two identical IDs appearing when executing seatunnel.sh -l as the job resumes (#5191) +- [Zeta] Fix SinkFlowLifeCycle without init lastCommitInfo (#5152) +- [Zeta] Fix MultipleTableJobConfigParser ignore env option (#5067) +- [Zeta] Fix checkpoint error report without msg (#5137) +- [Zeta] [REST-API] Fix class not found exception (#5063) +- [Zeta] [REST-API] Fix browser long type intercept (#5267) +- [Zeta] Fix task state memory leak (#5139) +- [Zeta] Fix when execute seatunnel.sh, console may print error (#4984) +- [Zeta] Fixed the issue of duplicated metrics caused by job fault tolerance or restore (#5214) +- [Zeta] Avoid Redundant Job Submissions by Checking Job Status (#5229) +- [Zeta] Fix the serialization issue of GetMetricsOperation during multi-node operation (#5206) +- [Zeta] Fix after the savepoint job is restored, the checkpoint file cannot be generated #4985 (#5051) +- [Zeta] Fixed multi-table job data loss and latency issues (#149) (#5031) +- [Zeta] Fix the HdfsStorage can not delete checkpoint file #5046 (#5054) +- [Zeta] Fix if the initialization task throws an error and the task can not return to master (#4888) + +### TEST +- [Test] Split updated modules integration test for part 5 (#5208) +- [Test] Add code style check when docs changed (#5183) +- [Test] Turn on the testSavepoint test (#5199) +- [Test] Fix the CI Job name error (#5032) +- [Test] [E2E] Fix testGetErrorInfo case error (#5282) +- [Test] [Kafka] Kafka e2e error (#5200) +- [Test] [MongoDB-CDC] Fix mongodb cdc e2e instability (#5128) +- [Test] Declare files that will always have UNIX line endings on checkout (#5221) +- [Test] Fixed a minor bug (#5274) +- [Test] Fixed IMap file storage e2e bug (#5237) +- [Test] Split updated modules integration test for part 4 (#5028) +- [Test] Fix zeta ci error (#5254) + +### Others +- Update the project description (#4967) +- Update version to 2.3.3-SNAPSHOT (#5043) +- Fix com.google.common.base.Preconditions to seatunnel shade one (#5284) +- Fix style error (#5280) +- Fix code style (#5092) ## Improve -- [Improve][Connector-V2][Jdbc-Source] Support for Decimal types as splict keys (#4634) - ### Core - -- [Core] [Spark] Push transform operation from Spark Driver to Executors (#4503) -- [Core] [Starter] Optimize code structure & remove redundant code (#4525) -- [Core] [Translation] [Flink] Optimize code structure & remove redundant code (#4527) -- [Core] [Starter] Add check of sink and source config to avoid null pointer exception. (#4734) - -### Connector-V2 - -- [Connector-V2] [CDC] Improve startup.mode/stop.mode options (#4360) -- [Connector-V2] [CDC] Optimize jdbc fetch-size options (#4352) -- [Connector-V2] [CDC] Fix chunk start/end parameter type error (#4777) -- [Connector-V2] [SQLServer] Fix sqlserver catalog (#4441) -- [Connector-V2] [StarRocks] Improve StarRocks Serialize Error Message (#4458) -- [Connector-V2] [Jdbc] add the log for sql and update some style (#4475) -- [Connector-V2] [Jdbc] Fix the table name is not automatically obtained when multiple tables (#4514) -- [Connector-V2] [S3 & Kafka] Delete unavailable S3 & Kafka Catalogs (#4477) -- [Connector-V2] [Pulsar] Support Canal Format -- [Connector-V2] [CDC base] Implement Sample-based Sharding Strategy with Configurable Sampling Rate (#4856) -- [Connector-V2] [SelectDB] Add a jobId to the selectDB label to distinguish between tasks (#4864) -- [Connector-V2] [Doris] Add a jobId to the doris label to distinguish between tasks (#4839) (#4853) -- [Connector-v2] [Mongodb]Refactor mongodb connector (#4620) -- [Connector-v2] [Jdbc] Populate primary key when jdbc sink is created using CatalogTable (#4755) -- [Connector-v2] [Neo4j] Supports neo4j sink batch write mode (#4835) -- [Transform-V2] Optimize SQL Transform package and Fix Spark type conversion bug of transform (#4490) +- [API & Zeta] Using connector custom serializer encode/decode states (#5238) +- [API]Improve savemode api (#4767) + +### Connector +- [CDC Base] Add some split parameters to the optionRule (#5161) +- [CDC Base] Support exactly-once of cdc and fix the BinlogOffset comparing bug (#5057) +- [CDC Base] Add a fast sampling method that supports character types (#5179) +- [CDC Base] Change driver scope to provider (#5002) +- [CDC Base] Remove driver for cdc connector (#4952) +- [Connector-V2] [File] Optimize files commit order (#5045) +- [Connector-V2] [HiveSink] Remove drop partition when abort (#4940) +- [Connector-V2] [Jdbc] Remove scheduler in JDBC sink #4736 (#5168) +- [Connector-V2] [Jdbc] Check url not null throw friendly message (#5097) +- [Connector-V2] [Jdbc] Support for Decimal types as splict keys (#4634) +- [Connector-V2] [Mongodb] Sink support transaction update/writing (#5034) +- [Connector-V2] [Mongodb] Optimize reading logic (#5001) +- [Connector-V2] [Neo4j] Supports neo4j sink batch write and update docs (#4841) +- [Connector-V2] [OSS-Jindo] Optimize jindo oss connector (#4964) +- [Connector-V2] [Tablestore sink] Remove scheduler in Tablestore sink (#5272) + +### Transform +- [Transform-V2] [SQL] Support use catalogTableName as SQL expression (#5273) ### CI - -- [CI] Fix error repository name in ci config files (#4795) +- [CI/CD] Remove 'paths-ignore', enable the code style check for markdown files (#5286) ### Zeta(ST-Engine) - -- [Zeta] Support run the server through daemon mode (#4161) -- [Zeta] Change ClassLoader To Improve the SDK compatibility of the client (#4447) -- [Zeta] Client Support Async Submit Job (#4456) -- [Zeta] Add more detailed log output. (#4446) -- [Zeta] Improve seatunnel-cluster.sh (#4435) -- [Zeta] Reduce CPU Cost When Task Not Ready (#4479) -- [Zeta] Add parser log (#4485) -- [Zeta] Remove redundant code (#4489) -- [Zeta] Remove redundancy code in validateSQL (#4506) -- [Zeta] Improve JobMetrics fetch performance (#4467) -- [Zeta] Reduce the operation count of imap_running_job_metrics (#4861) -- [Zeta] Speed up listAllJob function (#4852) -- [Zeta] async execute checkpoint trigger and other block method (#4846) -- [Zeta] Reduce the number of IMAPs used by checkpointIdCounter (#4832) -- [Zeta] Cancel pipeline add retry to avoid cancel failed. (#4792) -- [Zeta] Improve Zeta operation max count and ignore NPE (#4787) -- [Zeta] Remove serialize(deserialize) cost when use shuffle action (#4722) +- [Zeta] Upgrade the key log output level (#4993) +- [Zeta] Don't trigger handleSaveMode when restore (#5192) +- [Zeta] Move checkpoint notify complete in checkpoint stage (#5185) +- [Zeta] Add sleep for Task to reduce CPU cost (#5117) +- [Zeta] Add an interface for batch retrieval of JobMetrics (#4576) +- [Zeta] Move imap storage file dependency packages to submodules (#5218) +- [Zeta] Improve CheckpointCoordinator log error when report error from task (#178) (#5134) +- [Zeta] Improve CheckpointCoordinator notify complete when restore (#5136) ## Feature -### Core - -- [Core] [API] Support convert strings as List<T> option (#4362) -- [Core] [API] Add copy method to Catalog codes (#4414) -- [Core] [API] Add options check before create source and sink and transform in FactoryUtil (#4424) -- [Core] [Shade] Add guava shade module (#4358) - -### Connector-V2 - -- [Connector-V2] [CDC] [SQLServer] Support multi-table read (#4377) -- [Connector-V2] [Kafka] Kafka source supports data deserialization failure skipping (#4364) -- [Connector-V2] [Jdbc] [TiDB] Add TiDB catalog (#4438) -- [Connector-V2] [File] Add file excel sink and source (#4164) -- [Connector-v2] [Snowflake] Add Snowflake Source&Sink connector (#4470) -- [Connector-V2] [Pular] support read format for pulsar (#4111) -- [Connector-V2] [Paimon] Introduce paimon connector (#4178) -- [Connector V2] [Cassandra] Expose configurable options in Cassandra (#3681) -- [Connector V2] [Jdbc] Supports GEOMETRY data type for PostgreSQL (#4673) -- [Transform-V2] Add UDF SPI and an example implement for SQL Transform plugin (#4392) -- [Transform-V2] Support copy field list (#4404) -- [Transform-V2] Add support CatalogTable for FieldMapperTransform (#4423) -- [Transform-V2] Add CatalogTable support for ReplaceTransform (#4411) -- [Transform-V2] Add Catalog support for FilterRowKindTransform (#4420) -- [Transform-V2] Add support CatalogTable for FilterFieldTransform (#4422) -- [Transform-V2] Add catalog support for SQL Transform plugin (#4819) +### Connector +- [Connector-V2] [CDC-base] Support tables without primary keys (with unique keys) (#163) (#5150) +- [Connector-V2] [CDC-base] Support disable/enable exactly once for INITIAL (#4921) +- [Connector-V2] [CDC-base] Change the time zone to the default time zone (#5030) +- [Connector-V2] [CDC-base] Support string type shard fields (#5147) +- [Connector-V2] [Clickhouse] Add clickhouse connector time zone key,default system time zone (#5078) +- [Connector-V2] [Clickhouse] Clickhouse writes with checkpoints (#4999) +- [Connector-V2] [Doris] Add doris factory type convertor (#5061) +- [Connector-V2] [File] Add config of 'file_filter_pattern', which used for filtering files (#5153) +- [Connector-V2] [File] Add cos source&sink (#4979) +- [Connector-V2] [Hive] Hive add option because web need (#5154) +- [Connector-V2] [Jdbc] Add JDBC Catalog auto create table (#4917) +- [Connector-V2] [Jdbc] Add oceanbase dialect factory (#4989) +- [Connector-V2] [Jdbc] JDBC source support string type as partition key (#4947) +- [Connector-V2] [Jdbc-DM] Add DM upsert support (#5073) +- [Connector-V2] [Kafka] Support read debezium format message from kafka (#5066) +- [Connector-V2] [Kafka] Connector-kafka source support data conversion extracted by kafka connect source (#4516) +- [Connector-V2] [Mongodb] Mongodb support cdc sink (#4833) +- [Connector-V2] [Mongodb-CDC] Support source mongodb cdc (#4923) +- [Connector-V2] [Mysql-CDC] Conversion of tinyint(1) to bool is supported (#5105) +- [Connector-V2] [Redis] Support redis to set expiration time (#4975) ### Zeta(ST-Engine) +- [Zeta] [CDC-base] Support schema evolution framework(DDL) (#5125) +- [Zeta] [REST-API] Add REST API To Submit Job (#5107) +- [Zeta] Checkpoint support hdfs ha mode (#4942) +- [Zeta] The expiration time of a historical Job can be config (#5180) +- [Zeta] Support history service record job execute error (#5114) +- [Zeta] Configuration files support user variable replacement (#4969) -- [Zeta] Support for mixing Factory and Plugin SPI (#4359) -- [Zeta] Add get running job info by jobId rest api (#4140) -- [Zeta] Add REST API To Get System Monitoring Information (#4315) -- [Transform V2 & Zeta] Make SplitTransform Support CatalogTable And CatalogTable Evolution (#4396) -- [Zeta] Move driver into lib directory and change operation count (#4845) -- [Zeta] Add Metaspace size default value to config file (#4848) -- [Zeta] Reduce the frequency of fetching data from imap (#4851) -- [Zeta] Add OSS support for Imap storage to cluster-mode type (#4683) -- [Zeta] Improve local mode startup request ports (#4660) +### Transform +- [Transform-V2] [SQL] Support 'select *' and 'like' clause for SQL Transform plugin (#4991) + +### CI +- [CI] Update action config to support run CI on fork repo (#5065) +- [E2E] [FtpFile] Add ftp file e2e test case (#4647) ## Docs -- [Docs] Optimizes part of the Doris and SelectDB connector documentation (#4365) -- [Docs] Fix docs code style (#4368) -- [Docs] Update jdbc doc and kafka doc (#4380) -- [Docs] Fix max_retries default value is 0. (#4383) -- [Docs] Fix markdown syntax (#4426) -- [Docs] Fix Kafka Doc Error Config Key "kafka." (#4427) -- [Docs] Add Transform to Quick Start v2 (#4436) -- [Docs] Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (#4788) -- [Docs] Fix Mysql sink format doc (#4800) -- [Docs] Add the generate sink sql parameter for the jdbc sink document (#4797) -- [Docs] Add the generate sink sql parameter And example (#4769) -- [Docs] Redshift add defaultRowFetchSize (#4616) -- [Docs] Refactor connector-v2 docs using unified format Mysql (#4590) -- [Docs] Add Value types in Java to Schema features (#5087) +- [Connector-V2][Clickhouse] Reconstruct the clickhouse connector doc (#5085) +- [Connector-V2][Doris] Reconstruct the Doris connector document (#4903) +- [Connector-V2][HDFS] Refactor connector-v2 docs using unified format HDFS (#4871) +- [Connector-V2][Http] Reconstruct the Http connector document (#4962) +- [Connector-V2][Hudi] Reconstruct the Hudi connector document (#4905) +- [Connector-V2][Jdbc] Change the line boundary store value type to BigDecimal (#4900) +- [Connector-V2][Jdbc-DB2、Jdbc-Vertica] Improve DB2 Source Vertica Source & DB2 Sink Vertica Sink document (#5102) +- [Connector-V2][Jdbc-Oracle] Add JDBC Oracle Documentation (#5239) +- [Connector-V2][Kafka] Reconstruct the kafka connector document (#4778) +- [Connector-V2][MyHours] Reconstruct the MyHours connector document (#5129) +- [Connector-V2][PostgreSQL] Refactor connector-v2 docs using unified format PostgreSQL #4590 (#4757) +- [Connector-V2][S3File] Improve S3File Source & S3File Sink document (#5101) +- [Connector-V2][SelectDB-Cloud] Reconstruct the SelectDB-Cloud connector document (#5130) +- [Connector-V2][StarRocks] Reconstruct the StarRocks connector document (#5132) +- [Connector-V2][StarRocks] StarRocks `nodeUrls` property name fix (#4951) +- [README] Remove useless github workflow, and adjust description of 'engineering structure' (#4305) +- [Zeta] Add savepoint doc (#5081) +- Update bug-report.yml (#5160) +- Add Value types in Java to Schema feature (#5087) +- Documentation and partial word optimization (#4936) +- Change repository name from incubator-seatunnel to seatunnel (#4868) +- Update .asf.yaml (#5242) +- Update the Readme (#4968) +- Set Up with Kubernetes, dockerfile document error in constructing docker image (#5022) +- Fix Dockerfile and seatunnel-flink.yaml in Set Up with Kubernetes (#4793) +- Change the transform website url (#4954) +- Update quick-start-seatunnel-engine.md (#4943) +- Remove `incubating` keyword in document (#5257)
