This is an automated email from the ASF dual-hosted git repository.
leonard pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/flink-cdc.git
The following commit(s) were added to refs/heads/master by this push:
new ecaf82e43 [build] Update version to 3.5-SNAPSHOT
ecaf82e43 is described below
commit ecaf82e435052570d78fd92a41b195b287504079
Author: Kunni <[email protected]>
AuthorDate: Tue Apr 29 10:43:26 2025 +0800
[build] Update version to 3.5-SNAPSHOT
This closes #4008
---
Dockerfile | 2 +-
docs/config.toml | 5 +-
pom.xml | 2 +-
tools/mig-test/.gitignore | 8 -
tools/mig-test/README.md | 38 ---
tools/mig-test/conf/docker-compose.yml | 24 --
tools/mig-test/conf/pipeline-route.yaml | 41 ----
tools/mig-test/datastream/README.md | 11 -
tools/mig-test/datastream/compile_jobs.rb | 26 ---
.../datastream/datastream-3.2.0/.gitignore | 38 ---
tools/mig-test/datastream/datastream-3.2.0/pom.xml | 255 ---------------------
.../src/main/java/DataStreamJob.java | 50 ----
.../datastream/datastream-3.2.1/.gitignore | 38 ---
tools/mig-test/datastream/datastream-3.2.1/pom.xml | 255 ---------------------
.../src/main/java/DataStreamJob.java | 50 ----
.../datastream/datastream-3.3.0/.gitignore | 38 ---
tools/mig-test/datastream/datastream-3.3.0/pom.xml | 255 ---------------------
.../src/main/java/DataStreamJob.java | 50 ----
.../datastream/datastream-3.4-SNAPSHOT/.gitignore | 38 ---
.../datastream/datastream-3.4-SNAPSHOT/pom.xml | 255 ---------------------
.../src/main/java/DataStreamJob.java | 50 ----
tools/mig-test/datastream/run_migration_test.rb | 124 ----------
tools/mig-test/misc/patch_flink_conf.rb | 36 ---
tools/mig-test/prepare_libs.rb | 100 --------
tools/mig-test/run_migration_test.rb | 146 ------------
25 files changed, 5 insertions(+), 1930 deletions(-)
diff --git a/Dockerfile b/Dockerfile
index 7653e551b..fc050ad52 100644
--- a/Dockerfile
+++ b/Dockerfile
@@ -17,7 +17,7 @@
FROM flink
-ARG FLINK_CDC_VERSION=3.4-SNAPSHOT
+ARG FLINK_CDC_VERSION=3.5-SNAPSHOT
ARG PIPELINE_DEFINITION_FILE
RUN mkdir -p /opt/flink-cdc
diff --git a/docs/config.toml b/docs/config.toml
index 95fb76993..56236fa01 100644
--- a/docs/config.toml
+++ b/docs/config.toml
@@ -34,11 +34,11 @@ pygmentsUseClasses = true
# where we change the version for the complete docs when forking of a release
# branch etc.
# The full version string as referenced in Maven (e.g. 1.2.1)
- Version = "3.4-SNAPSHOT"
+ Version = "3.5-SNAPSHOT"
# For stable releases, leave the bugfix version out (e.g. 1.2). For snapshot
# release this should be the same as the regular version
- VersionTitle = "3.4-SNAPSHOT"
+ VersionTitle = "3.5-SNAPSHOT"
# The branch for this version of Apache Flink CDC
Branch = "master"
@@ -58,6 +58,7 @@ pygmentsUseClasses = true
]
PreviousDocs = [
+ ["3.4", "https://nightlies.apache.org/flink/flink-cdc-docs-release-3.4";],
["3.3", "https://nightlies.apache.org/flink/flink-cdc-docs-release-3.3";],
["3.2", "https://nightlies.apache.org/flink/flink-cdc-docs-release-3.2";],
["3.1", "https://nightlies.apache.org/flink/flink-cdc-docs-release-3.1";]
diff --git a/pom.xml b/pom.xml
index 0d700efa8..31728bbba 100644
--- a/pom.xml
+++ b/pom.xml
@@ -59,7 +59,7 @@ limitations under the License.
</scm>
<properties>
- <revision>3.4-SNAPSHOT</revision>
+ <revision>3.5-SNAPSHOT</revision>
<scala.binary.version>2.12</scala.binary.version>
<project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
<!-- Enforce single fork execution due to heavy mini cluster use in
the tests -->
diff --git a/tools/mig-test/.gitignore b/tools/mig-test/.gitignore
deleted file mode 100644
index d0ca45c15..000000000
--- a/tools/mig-test/.gitignore
+++ /dev/null
@@ -1,8 +0,0 @@
-*.sql
-savepoints/**
-cdc-versions/**
-cache/**
-.idea/**
-Gemfile.lock
-/logs/
-conf/temp.yaml
\ No newline at end of file
diff --git a/tools/mig-test/README.md b/tools/mig-test/README.md
deleted file mode 100644
index 39925cb95..000000000
--- a/tools/mig-test/README.md
+++ /dev/null
@@ -1,38 +0,0 @@
-# Flink CDC Migration Test Utilities
-
-## Pipeline Jobs
-### Preparation
-
-1. Install Ruby (macOS has embedded it by default)
-2. (Optional) Run `gem install terminal-table` for better display
-
-### Compile snapshot CDC versions
-3. Set `CDC_SOURCE_HOME` to the root directory of the Flink CDC git repository
-4. Go to `tools/mig-test` and run `ruby prepare_libs.rb` to download released
/ compile snapshot CDC versions
-
-### Run migration tests
-5. Enter `conf/` and run `docker compose up -d` to start up test containers
-6. Set `FLINK_HOME` to the home directory of Flink
-7. Go back to `tools/mig-test` and run `ruby run_migration_test.rb` to start
testing
-
-### Result
-The migration result will be displayed in the console like this:
-
-```
-+--------------------------------------------------------------------+
-| Migration Test Result |
-+--------------+-------+-------+-------+--------------+--------------+
-| | 3.0.0 | 3.0.1 | 3.1.0 | 3.1-SNAPSHOT | 3.2-SNAPSHOT |
-| 3.0.0 | ❓ | ❓ | ❌ | ✅ | ✅ |
-| 3.0.1 | | ❓ | ❌ | ✅ | ✅ |
-| 3.1.0 | | | ✅ | ❌ | ❌ |
-| 3.1-SNAPSHOT | | | | ✅ | ✅ |
-| 3.2-SNAPSHOT | | | | | ✅ |
-+--------------+-------+-------+-------+--------------+--------------+
-```
-
-> ✅ - Compatible, ❌ - Not compatible, ❓ - Target version doesn't support
`--from-savepoint`
-
-## DataStream Jobs
-
-See `datastream/README.md`.
diff --git a/tools/mig-test/conf/docker-compose.yml
b/tools/mig-test/conf/docker-compose.yml
deleted file mode 100644
index b195a59f6..000000000
--- a/tools/mig-test/conf/docker-compose.yml
+++ /dev/null
@@ -1,24 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-version: '3'
-services:
- mysql:
- image: mysql:8.0
- environment:
- MYSQL_ALLOW_EMPTY_PASSWORD: "true"
- MYSQL_DATABASE: fallen
- ports:
- - 3306:3306
diff --git a/tools/mig-test/conf/pipeline-route.yaml
b/tools/mig-test/conf/pipeline-route.yaml
deleted file mode 100644
index 0054d4d18..000000000
--- a/tools/mig-test/conf/pipeline-route.yaml
+++ /dev/null
@@ -1,41 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-source:
- type: mysql
- name: MySQL Source
- hostname: localhost
- port: 3306
- username: root
- password: ""
- tables: fallen.\.*
- server-id: 5400-5500
- server-time-zone: UTC
-
-sink:
- type: values
- name: Values Sink
-
-pipeline:
- name: ${PIPELINE_NAME}
- parallelism: 1
-
-transform:
- - source-table: fallen.\.*
- projection: \*, 'extras' AS EXTRAS
-
-route:
- - source-table: fallen.\.*
- sink-table: fallen.terminus
diff --git a/tools/mig-test/datastream/README.md
b/tools/mig-test/datastream/README.md
deleted file mode 100644
index e4d639b01..000000000
--- a/tools/mig-test/datastream/README.md
+++ /dev/null
@@ -1,11 +0,0 @@
-# Flink CDC MigrationTestUtils
-
-## DataStream Jobs
-### Preparation
-
-1. Install Ruby (macOS has embedded it by default)
-2. (Optional) Run `gem install terminal-table` for better display
-
-### Compile DataStream Jobs
-3. Go to `tools/mig-test/datastream` and run `ruby compile_jobs.rb` to compile
dummy DataStream jobs with specific version tags
-4. Then, run `ruby run_migration_test.rb` to start testing
\ No newline at end of file
diff --git a/tools/mig-test/datastream/compile_jobs.rb
b/tools/mig-test/datastream/compile_jobs.rb
deleted file mode 100644
index 2326a4540..000000000
--- a/tools/mig-test/datastream/compile_jobs.rb
+++ /dev/null
@@ -1,26 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-JOB_VERSIONS = %w[3.2.0 3.2.1 3.3.0 3.4-SNAPSHOT]
-
-JOB_VERSIONS.each do |version|
- puts "Compiling DataStream job for CDC #{version}"
- system "cd datastream-#{version} && mvn clean package -DskipTests"
-end
-
-puts 'Done'
\ No newline at end of file
diff --git a/tools/mig-test/datastream/datastream-3.2.0/.gitignore
b/tools/mig-test/datastream/datastream-3.2.0/.gitignore
deleted file mode 100644
index 5ff6309b7..000000000
--- a/tools/mig-test/datastream/datastream-3.2.0/.gitignore
+++ /dev/null
@@ -1,38 +0,0 @@
-target/
-!.mvn/wrapper/maven-wrapper.jar
-!**/src/main/**/target/
-!**/src/test/**/target/
-
-### IntelliJ IDEA ###
-.idea/modules.xml
-.idea/jarRepositories.xml
-.idea/compiler.xml
-.idea/libraries/
-*.iws
-*.iml
-*.ipr
-
-### Eclipse ###
-.apt_generated
-.classpath
-.factorypath
-.project
-.settings
-.springBeans
-.sts4-cache
-
-### NetBeans ###
-/nbproject/private/
-/nbbuild/
-/dist/
-/nbdist/
-/.nb-gradle/
-build/
-!**/src/main/**/build/
-!**/src/test/**/build/
-
-### VS Code ###
-.vscode/
-
-### Mac OS ###
-.DS_Store
\ No newline at end of file
diff --git a/tools/mig-test/datastream/datastream-3.2.0/pom.xml
b/tools/mig-test/datastream/datastream-3.2.0/pom.xml
deleted file mode 100644
index b624439ed..000000000
--- a/tools/mig-test/datastream/datastream-3.2.0/pom.xml
+++ /dev/null
@@ -1,255 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.flink</groupId>
- <artifactId>datastream-job</artifactId>
- <version>3.2.0</version>
- <packaging>jar</packaging>
-
- <properties>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <flink.version>1.18.1</flink.version>
- <flink.cdc.version>3.2.0</flink.cdc.version>
- <debezium.version>1.9.8.Final</debezium.version>
- <scala.binary.version>2.12</scala.binary.version>
- <slf4j.version>2.0.13</slf4j.version>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-base</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
-
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <!-- Checked the dependencies of the Flink project and below is a
feasible reference. -->
- <!-- Use flink shaded guava 18.0-13.0 for flink 1.13 -->
- <!-- Use flink shaded guava 30.1.1-jre-14.0 for flink-1.14 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.15 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.16 -->
- <!-- Use flink shaded guava 30.1.1-jre-16.1 for flink-1.17 -->
- <!-- Use flink shaded guava 31.1-jre-17.0 for flink-1.18 -->
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-shaded-guava</artifactId>
- <version>31.1-jre-17.0</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-debezium</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-base</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-mysql-cdc</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>io.debezium</groupId>
- <artifactId>debezium-connector-mysql</artifactId>
- <version>${debezium.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>${maven.compiler.source}</source>
- <target>${maven.compiler.target}</target>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <executions>
- <execution>
- <id>shade-flink</id>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <!-- Shading test jar have bug in some previous
version, so close this configuration here,
- see
https://issues.apache.org/jira/browse/MSHADE-284 -->
- <shadeTestJar>false</shadeTestJar>
-
<shadedArtifactAttached>false</shadedArtifactAttached>
-
<createDependencyReducedPom>true</createDependencyReducedPom>
- <dependencyReducedPomLocation>
-
${project.basedir}/target/dependency-reduced-pom.xml
- </dependencyReducedPomLocation>
- <filters combine.children="append">
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>module-info.class</exclude>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- <artifactSet>
- <includes>
- <include>io.debezium:debezium-api</include>
-
<include>io.debezium:debezium-embedded</include>
-
<include>io.debezium:debezium-core</include>
-
<include>io.debezium:debezium-ddl-parser</include>
-
<include>io.debezium:debezium-connector-mysql</include>
-
<include>org.apache.flink:flink-connector-debezium</include>
-
<include>org.apache.flink:flink-connector-mysql-cdc</include>
- <include>org.antlr:antlr4-runtime</include>
- <include>org.apache.kafka:*</include>
-
<include>mysql:mysql-connector-java</include>
-
<include>com.zendesk:mysql-binlog-connector-java</include>
- <include>com.fasterxml.*:*</include>
- <include>com.google.guava:*</include>
-
<include>com.esri.geometry:esri-geometry-api</include>
- <include>com.zaxxer:HikariCP</include>
- <!-- Include fixed version
30.1.1-jre-16.0 of flink shaded guava -->
-
<include>org.apache.flink:flink-shaded-guava</include>
- </includes>
- </artifactSet>
- <relocations>
- <relocation>
- <pattern>org.apache.kafka</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.apache.kafka
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.antlr</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.antlr
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.fasterxml</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.fasterxml
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.google
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.esri.geometry</pattern>
-
<shadedPattern>org.apache.flink.cdc.connectors.shaded.com.esri.geometry</shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.zaxxer</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.zaxxer
- </shadedPattern>
- </relocation>
- </relocations>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- <profiles>
- <profile>
- <id>java-8-target</id>
- <activation>
- <jdk>[1.8,11)</jdk>
- </activation>
- <properties>
- <java.version>1.8</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- <profile>
- <id>java-11-target</id>
- <activation>
- <jdk>[11,)</jdk>
- </activation>
- <properties>
- <java.version>11</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- </profiles>
-</project>
\ No newline at end of file
diff --git
a/tools/mig-test/datastream/datastream-3.2.0/src/main/java/DataStreamJob.java
b/tools/mig-test/datastream/datastream-3.2.0/src/main/java/DataStreamJob.java
deleted file mode 100644
index bfaa2d529..000000000
---
a/tools/mig-test/datastream/datastream-3.2.0/src/main/java/DataStreamJob.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.flink.cdc.connectors.mysql.source.MySqlSource;
-import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
-import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
-import org.apache.flink.api.common.eventtime.WatermarkStrategy;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-
-public class DataStreamJob {
-
- public static void main(String[] args) throws Exception {
- MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
- .hostname("localhost")
- .port(3306)
- .databaseList("fallen")
- .tableList("fallen.angel", "fallen.gabriel", "fallen.girl")
- .startupOptions(StartupOptions.initial())
- .username("root")
- .password("")
- .deserializer(new JsonDebeziumDeserializationSchema())
- .serverTimeZone("UTC")
- .build();
-
- StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
- env.enableCheckpointing(3000);
-
- env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "MySQL
CDC Source")
- .uid("sql-source-uid")
- .setParallelism(1)
- .print()
- .setParallelism(1);
-
- env.execute();
- }
-}
diff --git a/tools/mig-test/datastream/datastream-3.2.1/.gitignore
b/tools/mig-test/datastream/datastream-3.2.1/.gitignore
deleted file mode 100644
index 5ff6309b7..000000000
--- a/tools/mig-test/datastream/datastream-3.2.1/.gitignore
+++ /dev/null
@@ -1,38 +0,0 @@
-target/
-!.mvn/wrapper/maven-wrapper.jar
-!**/src/main/**/target/
-!**/src/test/**/target/
-
-### IntelliJ IDEA ###
-.idea/modules.xml
-.idea/jarRepositories.xml
-.idea/compiler.xml
-.idea/libraries/
-*.iws
-*.iml
-*.ipr
-
-### Eclipse ###
-.apt_generated
-.classpath
-.factorypath
-.project
-.settings
-.springBeans
-.sts4-cache
-
-### NetBeans ###
-/nbproject/private/
-/nbbuild/
-/dist/
-/nbdist/
-/.nb-gradle/
-build/
-!**/src/main/**/build/
-!**/src/test/**/build/
-
-### VS Code ###
-.vscode/
-
-### Mac OS ###
-.DS_Store
\ No newline at end of file
diff --git a/tools/mig-test/datastream/datastream-3.2.1/pom.xml
b/tools/mig-test/datastream/datastream-3.2.1/pom.xml
deleted file mode 100644
index decbc9f0c..000000000
--- a/tools/mig-test/datastream/datastream-3.2.1/pom.xml
+++ /dev/null
@@ -1,255 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.flink</groupId>
- <artifactId>datastream-job</artifactId>
- <version>3.2.1</version>
- <packaging>jar</packaging>
-
- <properties>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <flink.version>1.19.2</flink.version>
- <flink.cdc.version>3.2.1</flink.cdc.version>
- <debezium.version>1.9.8.Final</debezium.version>
- <scala.binary.version>2.12</scala.binary.version>
- <slf4j.version>2.0.13</slf4j.version>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-base</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
-
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <!-- Checked the dependencies of the Flink project and below is a
feasible reference. -->
- <!-- Use flink shaded guava 18.0-13.0 for flink 1.13 -->
- <!-- Use flink shaded guava 30.1.1-jre-14.0 for flink-1.14 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.15 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.16 -->
- <!-- Use flink shaded guava 30.1.1-jre-16.1 for flink-1.17 -->
- <!-- Use flink shaded guava 31.1-jre-17.0 for flink-1.18 -->
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-shaded-guava</artifactId>
- <version>31.1-jre-17.0</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-debezium</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-base</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-mysql-cdc</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>io.debezium</groupId>
- <artifactId>debezium-connector-mysql</artifactId>
- <version>${debezium.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>${maven.compiler.source}</source>
- <target>${maven.compiler.target}</target>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <executions>
- <execution>
- <id>shade-flink</id>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <!-- Shading test jar have bug in some previous
version, so close this configuration here,
- see
https://issues.apache.org/jira/browse/MSHADE-284 -->
- <shadeTestJar>false</shadeTestJar>
-
<shadedArtifactAttached>false</shadedArtifactAttached>
-
<createDependencyReducedPom>true</createDependencyReducedPom>
- <dependencyReducedPomLocation>
-
${project.basedir}/target/dependency-reduced-pom.xml
- </dependencyReducedPomLocation>
- <filters combine.children="append">
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>module-info.class</exclude>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- <artifactSet>
- <includes>
- <include>io.debezium:debezium-api</include>
-
<include>io.debezium:debezium-embedded</include>
-
<include>io.debezium:debezium-core</include>
-
<include>io.debezium:debezium-ddl-parser</include>
-
<include>io.debezium:debezium-connector-mysql</include>
-
<include>org.apache.flink:flink-connector-debezium</include>
-
<include>org.apache.flink:flink-connector-mysql-cdc</include>
- <include>org.antlr:antlr4-runtime</include>
- <include>org.apache.kafka:*</include>
-
<include>mysql:mysql-connector-java</include>
-
<include>com.zendesk:mysql-binlog-connector-java</include>
- <include>com.fasterxml.*:*</include>
- <include>com.google.guava:*</include>
-
<include>com.esri.geometry:esri-geometry-api</include>
- <include>com.zaxxer:HikariCP</include>
- <!-- Include fixed version
30.1.1-jre-16.0 of flink shaded guava -->
-
<include>org.apache.flink:flink-shaded-guava</include>
- </includes>
- </artifactSet>
- <relocations>
- <relocation>
- <pattern>org.apache.kafka</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.apache.kafka
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.antlr</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.antlr
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.fasterxml</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.fasterxml
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.google
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.esri.geometry</pattern>
-
<shadedPattern>org.apache.flink.cdc.connectors.shaded.com.esri.geometry</shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.zaxxer</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.zaxxer
- </shadedPattern>
- </relocation>
- </relocations>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- <profiles>
- <profile>
- <id>java-8-target</id>
- <activation>
- <jdk>[1.8,11)</jdk>
- </activation>
- <properties>
- <java.version>1.8</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- <profile>
- <id>java-11-target</id>
- <activation>
- <jdk>[11,)</jdk>
- </activation>
- <properties>
- <java.version>11</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- </profiles>
-</project>
\ No newline at end of file
diff --git
a/tools/mig-test/datastream/datastream-3.2.1/src/main/java/DataStreamJob.java
b/tools/mig-test/datastream/datastream-3.2.1/src/main/java/DataStreamJob.java
deleted file mode 100644
index bfaa2d529..000000000
---
a/tools/mig-test/datastream/datastream-3.2.1/src/main/java/DataStreamJob.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.flink.cdc.connectors.mysql.source.MySqlSource;
-import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
-import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
-import org.apache.flink.api.common.eventtime.WatermarkStrategy;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-
-public class DataStreamJob {
-
- public static void main(String[] args) throws Exception {
- MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
- .hostname("localhost")
- .port(3306)
- .databaseList("fallen")
- .tableList("fallen.angel", "fallen.gabriel", "fallen.girl")
- .startupOptions(StartupOptions.initial())
- .username("root")
- .password("")
- .deserializer(new JsonDebeziumDeserializationSchema())
- .serverTimeZone("UTC")
- .build();
-
- StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
- env.enableCheckpointing(3000);
-
- env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "MySQL
CDC Source")
- .uid("sql-source-uid")
- .setParallelism(1)
- .print()
- .setParallelism(1);
-
- env.execute();
- }
-}
diff --git a/tools/mig-test/datastream/datastream-3.3.0/.gitignore
b/tools/mig-test/datastream/datastream-3.3.0/.gitignore
deleted file mode 100644
index 5ff6309b7..000000000
--- a/tools/mig-test/datastream/datastream-3.3.0/.gitignore
+++ /dev/null
@@ -1,38 +0,0 @@
-target/
-!.mvn/wrapper/maven-wrapper.jar
-!**/src/main/**/target/
-!**/src/test/**/target/
-
-### IntelliJ IDEA ###
-.idea/modules.xml
-.idea/jarRepositories.xml
-.idea/compiler.xml
-.idea/libraries/
-*.iws
-*.iml
-*.ipr
-
-### Eclipse ###
-.apt_generated
-.classpath
-.factorypath
-.project
-.settings
-.springBeans
-.sts4-cache
-
-### NetBeans ###
-/nbproject/private/
-/nbbuild/
-/dist/
-/nbdist/
-/.nb-gradle/
-build/
-!**/src/main/**/build/
-!**/src/test/**/build/
-
-### VS Code ###
-.vscode/
-
-### Mac OS ###
-.DS_Store
\ No newline at end of file
diff --git a/tools/mig-test/datastream/datastream-3.3.0/pom.xml
b/tools/mig-test/datastream/datastream-3.3.0/pom.xml
deleted file mode 100644
index 0abaf4df5..000000000
--- a/tools/mig-test/datastream/datastream-3.3.0/pom.xml
+++ /dev/null
@@ -1,255 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.flink</groupId>
- <artifactId>datastream-job</artifactId>
- <version>3.3.0</version>
- <packaging>jar</packaging>
-
- <properties>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <flink.version>1.19.2</flink.version>
- <flink.cdc.version>3.3.0</flink.cdc.version>
- <debezium.version>1.9.8.Final</debezium.version>
- <scala.binary.version>2.12</scala.binary.version>
- <slf4j.version>2.0.13</slf4j.version>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-base</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
-
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <!-- Checked the dependencies of the Flink project and below is a
feasible reference. -->
- <!-- Use flink shaded guava 18.0-13.0 for flink 1.13 -->
- <!-- Use flink shaded guava 30.1.1-jre-14.0 for flink-1.14 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.15 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.16 -->
- <!-- Use flink shaded guava 30.1.1-jre-16.1 for flink-1.17 -->
- <!-- Use flink shaded guava 31.1-jre-17.0 for flink-1.18 -->
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-shaded-guava</artifactId>
- <version>31.1-jre-17.0</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-debezium</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-base</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-mysql-cdc</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>io.debezium</groupId>
- <artifactId>debezium-connector-mysql</artifactId>
- <version>${debezium.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>${maven.compiler.source}</source>
- <target>${maven.compiler.target}</target>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <executions>
- <execution>
- <id>shade-flink</id>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <!-- Shading test jar have bug in some previous
version, so close this configuration here,
- see
https://issues.apache.org/jira/browse/MSHADE-284 -->
- <shadeTestJar>false</shadeTestJar>
-
<shadedArtifactAttached>false</shadedArtifactAttached>
-
<createDependencyReducedPom>true</createDependencyReducedPom>
- <dependencyReducedPomLocation>
-
${project.basedir}/target/dependency-reduced-pom.xml
- </dependencyReducedPomLocation>
- <filters combine.children="append">
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>module-info.class</exclude>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- <artifactSet>
- <includes>
- <include>io.debezium:debezium-api</include>
-
<include>io.debezium:debezium-embedded</include>
-
<include>io.debezium:debezium-core</include>
-
<include>io.debezium:debezium-ddl-parser</include>
-
<include>io.debezium:debezium-connector-mysql</include>
-
<include>org.apache.flink:flink-connector-debezium</include>
-
<include>org.apache.flink:flink-connector-mysql-cdc</include>
- <include>org.antlr:antlr4-runtime</include>
- <include>org.apache.kafka:*</include>
-
<include>mysql:mysql-connector-java</include>
-
<include>com.zendesk:mysql-binlog-connector-java</include>
- <include>com.fasterxml.*:*</include>
- <include>com.google.guava:*</include>
-
<include>com.esri.geometry:esri-geometry-api</include>
- <include>com.zaxxer:HikariCP</include>
- <!-- Include fixed version
30.1.1-jre-16.0 of flink shaded guava -->
-
<include>org.apache.flink:flink-shaded-guava</include>
- </includes>
- </artifactSet>
- <relocations>
- <relocation>
- <pattern>org.apache.kafka</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.apache.kafka
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.antlr</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.antlr
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.fasterxml</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.fasterxml
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.google
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.esri.geometry</pattern>
-
<shadedPattern>org.apache.flink.cdc.connectors.shaded.com.esri.geometry</shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.zaxxer</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.zaxxer
- </shadedPattern>
- </relocation>
- </relocations>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- <profiles>
- <profile>
- <id>java-8-target</id>
- <activation>
- <jdk>[1.8,11)</jdk>
- </activation>
- <properties>
- <java.version>1.8</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- <profile>
- <id>java-11-target</id>
- <activation>
- <jdk>[11,)</jdk>
- </activation>
- <properties>
- <java.version>11</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- </profiles>
-</project>
\ No newline at end of file
diff --git
a/tools/mig-test/datastream/datastream-3.3.0/src/main/java/DataStreamJob.java
b/tools/mig-test/datastream/datastream-3.3.0/src/main/java/DataStreamJob.java
deleted file mode 100644
index bfaa2d529..000000000
---
a/tools/mig-test/datastream/datastream-3.3.0/src/main/java/DataStreamJob.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.flink.cdc.connectors.mysql.source.MySqlSource;
-import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
-import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
-import org.apache.flink.api.common.eventtime.WatermarkStrategy;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-
-public class DataStreamJob {
-
- public static void main(String[] args) throws Exception {
- MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
- .hostname("localhost")
- .port(3306)
- .databaseList("fallen")
- .tableList("fallen.angel", "fallen.gabriel", "fallen.girl")
- .startupOptions(StartupOptions.initial())
- .username("root")
- .password("")
- .deserializer(new JsonDebeziumDeserializationSchema())
- .serverTimeZone("UTC")
- .build();
-
- StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
- env.enableCheckpointing(3000);
-
- env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "MySQL
CDC Source")
- .uid("sql-source-uid")
- .setParallelism(1)
- .print()
- .setParallelism(1);
-
- env.execute();
- }
-}
diff --git a/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/.gitignore
b/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/.gitignore
deleted file mode 100644
index 5ff6309b7..000000000
--- a/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/.gitignore
+++ /dev/null
@@ -1,38 +0,0 @@
-target/
-!.mvn/wrapper/maven-wrapper.jar
-!**/src/main/**/target/
-!**/src/test/**/target/
-
-### IntelliJ IDEA ###
-.idea/modules.xml
-.idea/jarRepositories.xml
-.idea/compiler.xml
-.idea/libraries/
-*.iws
-*.iml
-*.ipr
-
-### Eclipse ###
-.apt_generated
-.classpath
-.factorypath
-.project
-.settings
-.springBeans
-.sts4-cache
-
-### NetBeans ###
-/nbproject/private/
-/nbbuild/
-/dist/
-/nbdist/
-/.nb-gradle/
-build/
-!**/src/main/**/build/
-!**/src/test/**/build/
-
-### VS Code ###
-.vscode/
-
-### Mac OS ###
-.DS_Store
\ No newline at end of file
diff --git a/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/pom.xml
b/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/pom.xml
deleted file mode 100644
index 69d7643aa..000000000
--- a/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/pom.xml
+++ /dev/null
@@ -1,255 +0,0 @@
-<?xml version="1.0" encoding="UTF-8"?>
-<!--
-Licensed to the Apache Software Foundation (ASF) under one or more
-contributor license agreements. See the NOTICE file distributed with
-this work for additional information regarding copyright ownership.
-The ASF licenses this file to You under the Apache License, Version 2.0
-(the "License"); you may not use this file except in compliance with
-the License. You may obtain a copy of the License at
-
- http://www.apache.org/licenses/LICENSE-2.0
-
-Unless required by applicable law or agreed to in writing, software
-distributed under the License is distributed on an "AS IS" BASIS,
-WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-See the License for the specific language governing permissions and
-limitations under the License.
--->
-<project xmlns="http://maven.apache.org/POM/4.0.0";
- xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance";
- xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd";>
- <modelVersion>4.0.0</modelVersion>
-
- <groupId>org.apache.flink</groupId>
- <artifactId>datastream-job</artifactId>
- <version>3.4-SNAPSHOT</version>
- <packaging>jar</packaging>
-
- <properties>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- <flink.version>1.19.2</flink.version>
- <flink.cdc.version>3.4-SNAPSHOT</flink.cdc.version>
- <debezium.version>1.9.8.Final</debezium.version>
- <scala.binary.version>2.12</scala.binary.version>
- <slf4j.version>2.0.13</slf4j.version>
- <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
- </properties>
-
- <dependencies>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-streaming-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-java</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-base</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-clients</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
-
<artifactId>flink-table-planner_${scala.binary.version}</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-runtime</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-core</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-table-common</artifactId>
- <version>${flink.version}</version>
- </dependency>
- <!-- Checked the dependencies of the Flink project and below is a
feasible reference. -->
- <!-- Use flink shaded guava 18.0-13.0 for flink 1.13 -->
- <!-- Use flink shaded guava 30.1.1-jre-14.0 for flink-1.14 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.15 -->
- <!-- Use flink shaded guava 30.1.1-jre-15.0 for flink-1.16 -->
- <!-- Use flink shaded guava 30.1.1-jre-16.1 for flink-1.17 -->
- <!-- Use flink shaded guava 31.1-jre-17.0 for flink-1.18 -->
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-shaded-guava</artifactId>
- <version>31.1-jre-17.0</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-debezium</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-cdc-base</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>org.apache.flink</groupId>
- <artifactId>flink-connector-mysql-cdc</artifactId>
- <version>${flink.cdc.version}</version>
- </dependency>
- <dependency>
- <groupId>io.debezium</groupId>
- <artifactId>debezium-connector-mysql</artifactId>
- <version>${debezium.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-api</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- <dependency>
- <groupId>org.slf4j</groupId>
- <artifactId>slf4j-simple</artifactId>
- <version>${slf4j.version}</version>
- </dependency>
- </dependencies>
-
- <build>
- <plugins>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-compiler-plugin</artifactId>
- <configuration>
- <source>${maven.compiler.source}</source>
- <target>${maven.compiler.target}</target>
- </configuration>
- </plugin>
- <plugin>
- <groupId>org.apache.maven.plugins</groupId>
- <artifactId>maven-shade-plugin</artifactId>
- <executions>
- <execution>
- <id>shade-flink</id>
- <phase>package</phase>
- <goals>
- <goal>shade</goal>
- </goals>
- <configuration>
- <!-- Shading test jar have bug in some previous
version, so close this configuration here,
- see
https://issues.apache.org/jira/browse/MSHADE-284 -->
- <shadeTestJar>false</shadeTestJar>
-
<shadedArtifactAttached>false</shadedArtifactAttached>
-
<createDependencyReducedPom>true</createDependencyReducedPom>
- <dependencyReducedPomLocation>
-
${project.basedir}/target/dependency-reduced-pom.xml
- </dependencyReducedPomLocation>
- <filters combine.children="append">
- <filter>
- <artifact>*:*</artifact>
- <excludes>
- <exclude>module-info.class</exclude>
- <exclude>META-INF/*.SF</exclude>
- <exclude>META-INF/*.DSA</exclude>
- <exclude>META-INF/*.RSA</exclude>
- </excludes>
- </filter>
- </filters>
- <artifactSet>
- <includes>
- <include>io.debezium:debezium-api</include>
-
<include>io.debezium:debezium-embedded</include>
-
<include>io.debezium:debezium-core</include>
-
<include>io.debezium:debezium-ddl-parser</include>
-
<include>io.debezium:debezium-connector-mysql</include>
-
<include>org.apache.flink:flink-connector-debezium</include>
-
<include>org.apache.flink:flink-connector-mysql-cdc</include>
- <include>org.antlr:antlr4-runtime</include>
- <include>org.apache.kafka:*</include>
-
<include>mysql:mysql-connector-java</include>
-
<include>com.zendesk:mysql-binlog-connector-java</include>
- <include>com.fasterxml.*:*</include>
- <include>com.google.guava:*</include>
-
<include>com.esri.geometry:esri-geometry-api</include>
- <include>com.zaxxer:HikariCP</include>
- <!-- Include fixed version
30.1.1-jre-16.0 of flink shaded guava -->
-
<include>org.apache.flink:flink-shaded-guava</include>
- </includes>
- </artifactSet>
- <relocations>
- <relocation>
- <pattern>org.apache.kafka</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.apache.kafka
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>org.antlr</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.org.antlr
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.fasterxml</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.fasterxml
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.google</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.google
- </shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.esri.geometry</pattern>
-
<shadedPattern>org.apache.flink.cdc.connectors.shaded.com.esri.geometry</shadedPattern>
- </relocation>
- <relocation>
- <pattern>com.zaxxer</pattern>
- <shadedPattern>
-
org.apache.flink.cdc.connectors.shaded.com.zaxxer
- </shadedPattern>
- </relocation>
- </relocations>
- </configuration>
- </execution>
- </executions>
- </plugin>
- </plugins>
- </build>
- <profiles>
- <profile>
- <id>java-8-target</id>
- <activation>
- <jdk>[1.8,11)</jdk>
- </activation>
- <properties>
- <java.version>1.8</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- <profile>
- <id>java-11-target</id>
- <activation>
- <jdk>[11,)</jdk>
- </activation>
- <properties>
- <java.version>11</java.version>
- <maven.compiler.source>${java.version}</maven.compiler.source>
- <maven.compiler.target>${java.version}</maven.compiler.target>
- </properties>
- </profile>
- </profiles>
-</project>
\ No newline at end of file
diff --git
a/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/src/main/java/DataStreamJob.java
b/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/src/main/java/DataStreamJob.java
deleted file mode 100644
index bfaa2d529..000000000
---
a/tools/mig-test/datastream/datastream-3.4-SNAPSHOT/src/main/java/DataStreamJob.java
+++ /dev/null
@@ -1,50 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-import org.apache.flink.cdc.connectors.mysql.source.MySqlSource;
-import org.apache.flink.cdc.connectors.mysql.table.StartupOptions;
-import org.apache.flink.cdc.debezium.JsonDebeziumDeserializationSchema;
-import org.apache.flink.api.common.eventtime.WatermarkStrategy;
-import org.apache.flink.streaming.api.environment.StreamExecutionEnvironment;
-
-public class DataStreamJob {
-
- public static void main(String[] args) throws Exception {
- MySqlSource<String> mySqlSource = MySqlSource.<String>builder()
- .hostname("localhost")
- .port(3306)
- .databaseList("fallen")
- .tableList("fallen.angel", "fallen.gabriel", "fallen.girl")
- .startupOptions(StartupOptions.initial())
- .username("root")
- .password("")
- .deserializer(new JsonDebeziumDeserializationSchema())
- .serverTimeZone("UTC")
- .build();
-
- StreamExecutionEnvironment env =
StreamExecutionEnvironment.getExecutionEnvironment();
- env.enableCheckpointing(3000);
-
- env.fromSource(mySqlSource, WatermarkStrategy.noWatermarks(), "MySQL
CDC Source")
- .uid("sql-source-uid")
- .setParallelism(1)
- .print()
- .setParallelism(1);
-
- env.execute();
- }
-}
diff --git a/tools/mig-test/datastream/run_migration_test.rb
b/tools/mig-test/datastream/run_migration_test.rb
deleted file mode 100644
index 951d043da..000000000
--- a/tools/mig-test/datastream/run_migration_test.rb
+++ /dev/null
@@ -1,124 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-require 'pathname'
-require 'securerandom'
-
-WAITING_SECONDS = 20
-FLINK_HOME = ENV['FLINK_HOME']
-throw 'Unspecified `FLINK_HOME` environment variable.' if FLINK_HOME.nil?
-FLINK_HOME = Pathname.new(FLINK_HOME).realpath
-
-SOURCE_PORT = 3306
-DATABASE_NAME = 'fallen'
-TABLES = ['girl'].freeze
-
-def exec_sql_source(sql)
- `mysql -h 127.0.0.1 -P#{SOURCE_PORT} -uroot --skip-password -e "USE
#{DATABASE_NAME}; #{sql}"`
-end
-
-def extract_job_id(output)
- current_job_id = output.split("\n").filter { _1.start_with?('Job has been
submitted with JobID ') }.first&.split&.last
- raise StandardError, "Failed to submit Flink job. Output: #{output}" unless
current_job_id&.length == 32
- current_job_id
-end
-
-def put_mystery_data(mystery)
- exec_sql_source("REPLACE INTO girl(id, name) VALUES (17, '#{mystery}');")
-end
-
-def ensure_mystery_data(mystery)
- raise StandardError, 'Failed to get specific mystery string' unless `cat
#{FLINK_HOME}/log/*.out`.include? mystery
-end
-
-puts ' Waiting for source to start up...'
-next until exec_sql_source("SELECT '1';") == "1\n1\n"
-
-def test_migration_chore(from_version, to_version)
- TABLES.each do |table_name|
- exec_sql_source("DROP TABLE IF EXISTS #{table_name};")
- exec_sql_source("CREATE TABLE #{table_name} (ID INT NOT NULL, NAME
VARCHAR(17), PRIMARY KEY (ID));")
- end
-
- # Clear previous savepoints and logs
- `rm -rf savepoints`
-
- old_output = `#{FLINK_HOME}/bin/flink run -p 1 -c DataStreamJob --detached
datastream-#{from_version}/target/datastream-job-#{from_version}.jar`
- old_job_id = extract_job_id(old_output)
-
- puts "Submitted job at #{from_version} as #{old_job_id}"
-
- random_string_1 = SecureRandom.hex(8)
- put_mystery_data random_string_1
- sleep WAITING_SECONDS
- ensure_mystery_data random_string_1
-
- puts `#{FLINK_HOME}/bin/flink stop --savepointPath #{Dir.pwd}/savepoints
#{old_job_id}`
- savepoint_file = `ls savepoints`.split("\n").last
- new_output = `#{FLINK_HOME}/bin/flink run --fromSavepoint
#{Dir.pwd}/savepoints/#{savepoint_file} -p 1 -c DataStreamJob --detached
datastream-#{to_version}/target/datastream-job-#{to_version}.jar`
- new_job_id = extract_job_id(new_output)
-
- puts "Submitted job at #{to_version} as #{new_job_id}"
- random_string_2 = SecureRandom.hex(8)
- put_mystery_data random_string_2
- sleep WAITING_SECONDS
- ensure_mystery_data random_string_2
- puts `#{FLINK_HOME}/bin/flink cancel #{new_job_id}`
- true
-end
-
-def test_migration(from_version, to_version)
- puts "➡️ [MIGRATION] Testing migration from #{from_version} to
#{to_version}..."
- puts " with Flink #{FLINK_HOME}..."
- begin
- result = test_migration_chore from_version, to_version
- if result
- puts "✅ [MIGRATION] Successfully migrated from #{from_version} to
#{to_version}!"
- else
- puts "❌ [MIGRATION] Failed to migrate from #{from_version} to
#{to_version}..."
- end
- result
- rescue => e
- puts "❌ [MIGRATION] Failed to migrate from #{from_version} to
#{to_version}...", e
- false
- end
-end
-
-version_list = %w[3.2.0 3.2.1 3.3.0 3.4-SNAPSHOT]
-@failures = []
-
-new_version = version_list.last
-
-version_list.each do |old_version|
- puts "-> Testing migrating from #{old_version} to latest snapshot."
- puts 'Restarting cluster...'
- `#{FLINK_HOME}/bin/stop-cluster.sh`
- `rm -rf #{FLINK_HOME}/log/flink-*.out`
- puts 'Stopped cluster.'
- `#{FLINK_HOME}/bin/start-cluster.sh`
- puts 'Started cluster.'
-
- result = test_migration old_version, new_version
- @failures << [old_version, new_version] unless result
-end
-
-if @failures.any?
- puts 'Some migration to snapshot version tests failed. Details: '
- puts @failures
- abort 'Some migration to snapshot version tests failed.'
-end
diff --git a/tools/mig-test/misc/patch_flink_conf.rb
b/tools/mig-test/misc/patch_flink_conf.rb
deleted file mode 100644
index 6241a52c3..000000000
--- a/tools/mig-test/misc/patch_flink_conf.rb
+++ /dev/null
@@ -1,36 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-FLINK_HOME = ENV['FLINK_HOME']
-throw 'Unspecified `FLINK_HOME` environment variable.' if FLINK_HOME.nil?
-
-EXTRA_CONF = <<~EXTRACONF
-
-taskmanager.numberOfTaskSlots: 10
-parallelism.default: 4
-execution.checkpointing.interval: 300
-EXTRACONF
-
-if File.file?("#{FLINK_HOME}/conf/flink-conf.yaml")
- File.write("#{FLINK_HOME}/conf/flink-conf.yaml", EXTRA_CONF, mode: 'a+')
-else
- File.write("#{FLINK_HOME}/conf/config.yaml", EXTRA_CONF, mode: 'a+')
-end
-
-# MySQL connector is not provided
-`wget
https://repo1.maven.org/maven2/mysql/mysql-connector-java/8.0.27/mysql-connector-java-8.0.27.jar
-O #{FLINK_HOME}/lib/mysql-connector-java-8.0.27.jar`
\ No newline at end of file
diff --git a/tools/mig-test/prepare_libs.rb b/tools/mig-test/prepare_libs.rb
deleted file mode 100644
index 4aceac8d0..000000000
--- a/tools/mig-test/prepare_libs.rb
+++ /dev/null
@@ -1,100 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-CDC_SOURCE_HOME = ENV['CDC_SOURCE_HOME']
-throw 'Unspecified `CDC_SOURCE_HOME` environment variable.' if
CDC_SOURCE_HOME.nil?
-
-Dir.chdir(__dir__)
-
-def gen_version(tag)
- {
- tar:
"https://dlcdn.apache.org/flink/flink-cdc-#{tag}/flink-cdc-#{tag}-bin.tar.gz";,
- connectors: %W[
-
https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-mysql/#{tag}/flink-cdc-pipeline-connector-mysql-#{tag}.jar
-
https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-doris/#{tag}/flink-cdc-pipeline-connector-doris-#{tag}.jar
-
https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-starrocks/#{tag}/flink-cdc-pipeline-connector-starrocks-#{tag}.jar
-
https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-kafka/#{tag}/flink-cdc-pipeline-connector-kafka-#{tag}.jar
-
https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-paimon/#{tag}/flink-cdc-pipeline-connector-paimon-#{tag}.jar
-
https://repo1.maven.org/maven2/org/apache/flink/flink-cdc-pipeline-connector-values/#{tag}/flink-cdc-pipeline-connector-values-#{tag}.jar
- ]
- }
-end
-
-RELEASED_VERSIONS = {
- '3.2.0': gen_version('3.2.0'),
- '3.2.1': gen_version('3.2.1'),
- '3.3.0': gen_version('3.3.0'),
-}.freeze
-
-HEAD_VERSION = '3.4-SNAPSHOT'
-
-def download_or_get(link)
- `mkdir -p cache`
- file_name = "cache/#{File.basename(link)}"
- if File.exist? file_name
- puts "#{file_name} exists, skip download"
- return file_name
- end
- `wget #{link} -O #{file_name}`
- file_name
-end
-
-M2_REPO = '~/.m2/repository/org/apache/flink'
-FILES = %w[
- dist
- pipeline-connector-kafka
- pipeline-connector-mysql
- pipeline-connector-doris
- pipeline-connector-paimon
- pipeline-connector-starrocks
- pipeline-connector-values
-].freeze
-def download_released
- `rm -rf cdc-versions`
- RELEASED_VERSIONS.each do |version, links|
- `mkdir -p cdc-versions/#{version}`
- file_name = download_or_get(links[:tar])
- `tar --strip-components=1 -xzvf #{file_name} -C cdc-versions/#{version}`
- links[:connectors].each do |link|
- jar_file_name = download_or_get(link)
- `cp #{jar_file_name} cdc-versions/#{version}/lib/`
- end
- end
-end
-
-def compile_snapshot(version)
- puts "Trying to create #{version}"
- `mkdir -p cdc-versions/#{version}/lib`
- `cp -r #{CDC_SOURCE_HOME}/flink-cdc-dist/src/main/flink-cdc-bin/*
cdc-versions/#{version}/`
-
- puts 'Compiling snapshot version...'
- puts `cd #{CDC_SOURCE_HOME} && mvn clean install -DskipTests`
-
- FILES.each do |lib|
- if lib == 'dist'
- `cp
#{CDC_SOURCE_HOME}/flink-cdc-#{lib}/target/flink-cdc-#{lib}-#{version}.jar
cdc-versions/#{version}/lib/`
- else
- `cp
#{CDC_SOURCE_HOME}/flink-cdc-connect/flink-cdc-pipeline-connectors/flink-cdc-#{lib}/target/flink-cdc-#{lib}-#{version}.jar
cdc-versions/#{version}/lib/`
- end
- end
-end
-
-download_released
-compile_snapshot HEAD_VERSION
-
-puts 'Done'
diff --git a/tools/mig-test/run_migration_test.rb
b/tools/mig-test/run_migration_test.rb
deleted file mode 100644
index 07dba0180..000000000
--- a/tools/mig-test/run_migration_test.rb
+++ /dev/null
@@ -1,146 +0,0 @@
-#!/usr/bin/env ruby
-# frozen_string_literal: true
-
-# Licensed to the Apache Software Foundation (ASF) under one or more
-# contributor license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright ownership.
-# The ASF licenses this file to You under the Apache License, Version 2.0
-# (the "License"); you may not use this file except in compliance with
-# the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing, software
-# distributed under the License is distributed on an "AS IS" BASIS,
-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# See the License for the specific language governing permissions and
-# limitations under the License.
-
-require 'pathname'
-require 'securerandom'
-
-WAITING_SECONDS = 20
-FLINK_HOME = ENV['FLINK_HOME']
-throw 'Unspecified `FLINK_HOME` environment variable.' if FLINK_HOME.nil?
-FLINK_HOME = Pathname.new(FLINK_HOME).realpath
-
-SOURCE_PORT = 3306
-DATABASE_NAME = 'fallen'
-TABLES = ['girl'].freeze
-
-def exec_sql_source(sql)
- `mysql -h 127.0.0.1 -P#{SOURCE_PORT} -uroot --skip-password -e "USE
#{DATABASE_NAME}; #{sql}"`
-end
-
-def put_mystery_data(mystery)
- exec_sql_source("REPLACE INTO girl(id, name) VALUES (17, '#{mystery}');")
-end
-
-def ensure_mystery_data(mystery)
- raise StandardError, 'Failed to get specific mystery string' unless `cat
#{FLINK_HOME}/log/*.out`.include? mystery
-end
-
-def extract_job_id(output)
- current_job_id = output.split("\n").filter { _1.start_with?('Job ID: ')
}.first&.split&.last
- raise StandardError, "Failed to submit Flink job. Output: #{output}" unless
current_job_id&.length == 32
- current_job_id
-end
-
-puts ' Waiting for source to start up...'
-next until exec_sql_source("SELECT '1';") == "1\n1\n"
-
-def test_migration_chore(from_version, to_version)
- TABLES.each do |table_name|
- exec_sql_source("DROP TABLE IF EXISTS #{table_name};")
- exec_sql_source("CREATE TABLE #{table_name} (ID INT NOT NULL, NAME
VARCHAR(17), PRIMARY KEY (ID));")
- end
-
- # Clear previous savepoints and logs
- `rm -rf savepoints`
-
- # Prepare for current YAML file
- yaml_job_template_file = 'conf/pipeline-route.yaml'
- yaml_job_file = 'conf/temp.yaml'
- yaml_content =
File.open(yaml_job_template_file).read.gsub('${PIPELINE_NAME}',
- "Pipeline
Migration Job (#{from_version} -> #{to_version})")
- File.write(yaml_job_file, yaml_content)
-
- # Submit current pipeline job file
- submit_job_output = `bash ./cdc-versions/#{from_version}/bin/flink-cdc.sh
--flink-home #{FLINK_HOME} #{yaml_job_file}`
- puts " #{submit_job_output}"
- current_job_id = extract_job_id(submit_job_output)
-
- puts " Current Job ID: #{current_job_id}"
-
- # Verify if data sync works
- random_string_1 = SecureRandom.hex(8)
- put_mystery_data random_string_1
- sleep WAITING_SECONDS
- ensure_mystery_data random_string_1
-
- # Stop current job and create a savepoint
- puts `#{FLINK_HOME}/bin/flink stop --savepointPath #{Dir.pwd}/savepoints
#{current_job_id}`
- savepoint_file = `ls savepoints`.split("\n").last
-
- # Migrate to a newer CDC version
- puts " Submitting CDC jobs at #{to_version}..."
- submit_job_output = `bash ./cdc-versions/#{to_version}/bin/flink-cdc.sh
--from-savepoint #{Dir.pwd}/savepoints/#{savepoint_file}
--allow-nonRestored-state --flink-home #{FLINK_HOME} #{yaml_job_file}`
- puts "#{submit_job_output}"
- new_job_id = extract_job_id(submit_job_output)
-
- puts " Upgraded Job ID: #{new_job_id}"
-
- # Verify if data sync works
- puts "Submitted job at #{to_version} as #{new_job_id}"
- random_string_2 = SecureRandom.hex(8)
- put_mystery_data random_string_2
- sleep WAITING_SECONDS
- ensure_mystery_data random_string_2
- puts `#{FLINK_HOME}/bin/flink cancel #{new_job_id}`
- true
-end
-
-def test_migration(from_version, to_version)
- puts "➡️ [MIGRATION] Testing migration from #{from_version} to
#{to_version}..."
- puts " with Flink #{FLINK_HOME}..."
- begin
- result = test_migration_chore from_version, to_version
- if result
- puts "✅ [MIGRATION] Successfully migrated from #{from_version} to
#{to_version}!"
- else
- puts "❌ [MIGRATION] Failed to migrate from #{from_version} to
#{to_version}..."
- end
- result
- rescue => e
- puts "❌ [MIGRATION] Failed to migrate from #{from_version} to
#{to_version}...", e
- false
- end
-end
-
-version_list = case ARGV[0]
- when '1.19.2' then %w[3.2.0 3.2.1 3.3.0 3.4-SNAPSHOT]
- when '1.20.1' then %w[3.2.1 3.3.0 3.4-SNAPSHOT]
- else throw "Unknown Flink target version: #{ARGV[0]}."
- end
-
-@failures = []
-
-new_version = version_list.last
-
-version_list.each do |old_version|
- puts "-> Testing migrating from #{old_version} to latest snapshot."
- puts 'Restarting cluster...'
- `#{FLINK_HOME}/bin/stop-cluster.sh`
- puts 'Stopped cluster.'
- `#{FLINK_HOME}/bin/start-cluster.sh`
- puts 'Started cluster.'
-
- result = test_migration old_version, new_version
- @failures << "#{old_version} => #{new_version}" unless result
-end
-
-if @failures.any?
- puts 'Some migration to snapshot version tests failed. Details: '
- puts @failures
- abort 'Some migration to snapshot version tests failed.'
-end
