[
https://issues.apache.org/jira/browse/FLINK-41452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
peiyu updated FLINK-41452:
--------------------------
Description:
h1. Multi-Flink-version compatibility refactor
h2. 1. Background
h3. 1.1 The support matrix keeps shrinking
Flink CDC's Flink support matrix has been contracting: 3.1 supported 4 Flink
versions (1.16-1.19), from 3.5 only 1.19/1.20 remained, and 3.6/3.7 support
just 1.20 + 2.2. The marginal cost of adding one Flink version is "a full round
of build configuration plus profile surgery", and that cost curve directly
determines how wide the matrix can be.
h3. 1.2 Four structural problems with the current mechanism
The current mechanism (3.7 / 3.8-SNAPSHOT) is "single source tree + whole-tree
Maven profile switching + dual-FQCN shadow modules":
* *Profile fragmentation.* Generation switching (1.20 vs 2.2) is scattered
across the local profiles of 7 modules; the root pom's \{{flink2}} profile does
not even switch \{{flink.version}}; the runtime module needs a
property-activated \{{*:*}} exclusion profile to neutralize the default one;
test classpaths are hand-assembled via a \{{copy-flink2-guava}} hack.
* *The guava package-name trap.* 89+ files hardcode \{{import
org.apache.flink.shaded.guava31.*}}; the \{{flink-shaded-guava}} package prefix
evolves with Flink (1.20 -> guava31, 2.2 -> guava33). Under the flink2 build,
main modules compile against 31.1 while dist packages 33.4 (verified via
\{{dependency:tree}}), and at runtime we are saved only by the guava31
compatibility bundle retained in the Flink 2.2 distribution (FLINK-39288 /
flink-cdc PR #4339) - an implicit dependency with no contract behind it.
* *Shadow-class linkage risk without per-version verification.* The dist and
SQL connector fat jars contain 30+ \{{org.apache.flink.*}} shadow classes
(backports / copies). Linking correctness depends entirely on the shadowed
classes happening to be binary-compatible with the real Flink APIs, and there
is no systematic verification.
* *Main-code reflection into Flink internals.* \{{DataSinkWriterOperator}}
reflectively probes the 3-arg \{{SinkWriterOperator}} constructor and falls
back to the 4-arg one; the composer reflectively reads the private
\{{StreamExecutionEnvironment.configuration}} field. These are silent breakage
points on every Flink upgrade.
Two adjacent gaps: the external connector dependency matrix is incomplete (the
doris / starrocks / hudi external connectors are hardcoded to the 1.20 line,
which is "fake compatibility" under flink2 builds without an explicit
declaration), and source-connector e2e coverage is a single 1.20 CI job.
h3. 1.3 Prior art
Paimon and Fluss have converged on the same pattern: *common code compiled
against a per-generation baseline + one thin module per supported Flink minor +
one artifact jar per version*, and neither depends on \{{flink-shaded-guava}}
(both relocate guava into their own package space).
They differ in how the common layer is compiled: Paimon compiles it per
generation (\{{paimon-flink1-common}} / \{{paimon-flink2-common}}), while Fluss
compiles everything once against a single baseline and routes all differences
through Adapter overrides plus backports. This proposal follows the Paimon
shape: flink-cdc's legacy \{{SourceFunction}} ecosystem (the source connectors,
\{{DebeziumSourceFunction}}, the legacy branches of 8 TableSources) and the
2.x-new APIs (\{{With*Topology}}, \{{StateRecoveryOptions}}) present a
bidirectional sensitivity surface far larger than Fluss's (a young project with
no legacy); a single baseline would maximize shadow-class linkage risk, whereas
per-generation compilation links every line's shadowed classes against that
line's real Flink API.
h2. 2. Proposal: the target architecture
h3. 2.1 Three-layer compatibility structure (following Paimon)
{noformat}
flink-cdc-flink/ # Flink version compatibility layer
├── flink-cdc-flink1-common/ # generation-common layer (compiled
against 1.20.3):
│ # 2.x API backports + 1.20-line
Adapters
├── flink-cdc-flink2-common/ # generation-common layer (compiled
against 2.2.0):
│ # copies of APIs removed in 2.x +
2.2-line Adapters
├── flink-cdc-flink-1.20/ # version thin module (skeleton +
exact-version UTs;
│ # will host intra-generation minor
differences)
└── flink-cdc-flink-2.2/
{noformat}
* *The root pom is the only generation switch point.* The \{{flink1}} (default)
and \{{flink2}} profiles centralize all generation-scoped properties; modules
keep zero local profiles (the 7 modules' fragmented profiles are gone).
* *Version-sensitive APIs must go through Adapters.* Main code may not directly
reference Flink APIs that fork across generations. The two generations'
Adapters share FQCN and have fully identical self-declared member signatures
(contract in 4.2); the forks are digested inside the compatibility layer.
Enforced by a bytecode-scan guard (\{{ReflectionGuardTest}}) plus a review
checklist.
* *Guava self-governance.* A \{{flink-cdc-shaded-guava}} module relocates guava
to \{{org.apache.flink.cdc.shaded.guava.*}} (no version prefix, so the
guava31->33 prefix drift cannot repeat), the root pom switches to it globally,
and fat jars ship their own copy. This proceeds in synergy with FLINK-35071.
h3. 2.2 Artifact layout
||Artifact class||GAV shape||Basis||
|Intermediate modules (common / runtime / composer / cli / python / debezium /
source-connector code modules)|Single GAV (e.g. \{{flink-cdc-runtime:3.8.0}}),
deployed exactly once|Cross-generation jar-level comparison: bytecode
differences reduced to Flink annotation \{{@Retention}} differences (2 bytes,
zero runtime impact) - see 4.1|
|Pipeline distribution|\{{flink-cdc-dist-flink-1.20}} /
\{{flink-cdc-dist-flink-2.2}}|Shaded content is strongly generation-bound|
|9 SQL source-connector fat jars|\{{flink-sql-connector-mysql-cdc-flink-1.20}}
/ \{{-flink-2.2}}, etc.|Same|
|4 pipeline-connector fat jars (kafka, paimon, elasticsearch,
iceberg)|\{{flink-cdc-pipeline-connector-kafka-flink-1.20}} / \{{-flink-2.2}},
etc.|The shaded external connector lines fork across generations (kafka fat
jar: 2918 vs 4521 entries)|
|The other 11 pipeline-connector fat jars|Single GAV kept|Jar-level comparison
proves cross-generation content equivalence (annotation/metadata-only
differences, or byte-identical)|
*Invariant:* the repository no longer contains any artifact whose content
varies across Flink generations while its GAV carries no generation identity.
The Flink line is carried by the artifactId suffix and the version string stays
clean (\{{3.8.0}}) - consistent with Paimon (\{{paimon-flink-1.20:1.4.2}}) and
Fluss.
h3. 2.3 Compatibility promises
* \{{@Public}} APIs are untouched.
* DataStream API users' dependency coordinates do not change (the thin
single-GAV artifacts are usable on both generations - cross-generation
equivalence is verified).
* Zero runtime behavior change: every fat jar was compared class-by-class (name
+ CRC32) against the pre-refactor artifact of the corresponding Flink line;
class content is identical (a placeholder class aside).
* JDK 11 baseline, Maven 3.8.6+, single reactor - unchanged.
h2. 3. Implementation status
Implementation is complete on a feature branch: six phases, one commit each -
complete, independently reviewable change units. This is not a request for
permission to proceed, but a request to review the direction and the concrete
shape.
||Phase||Scope of the commit||
|1|Guava self-governance: a \{{flink-cdc-shaded-guava}} module relocating guava
to \{{org.apache.flink.cdc.shaded.guava.*}} (no version prefix) + a one-shot
import replacement across 89+ files + NOTICE/LICENSE updates|
|2|Build hygiene: the root pom becomes the single generation switch point; the
7 modules' fragmented local profiles, the neutralizing profiles and the
\{{copy-flink2-guava}} classpath hack are removed; starrocks/hudi flink2 output
gating with CI and README declarations|
|3|Compatibility layer reorg: the \{{flink-cdc-flink/}} aggregate with the two
generation-common modules (migrated and renamed, zero class-content change) and
the two version thin modules (skeleton + exact-version UTs), each thin module
pinned to its line-scoped Flink version so a future generation-baseline bump
cannot drag an older line along|
|4|Per-version dist produced from a single line-parameterized module:
\{{flink-cdc-dist}} shades the pipeline core and assembles the bin tarball, and
its artifactId carries the Flink line of the build
(\{{flink-cdc-dist-flink-${flink.line}}}), so every supported line gets its
distribution from one module with no per-line clones. The Yarn dist-jar lookup
derives the Flink line at runtime from the running jar's own identity
(\{{flink-cdc-dist-flink-<ver>-*.jar}} name parse) instead of a build-time
filtered resource - a single build-time value cannot serve two same-generation
releases (e.g. a future 2.2 + 2.3 produced by one \{{-Pflink2}} build), while
self-identifying jars always answer for themselves; unidentifiable classpath
shapes fail loudly. Release-script adaptation + transition-period legacy-named
artifacts; the dist shade-priority guard ITCase lives in the module and skips
itself on lines without the marker|
|5|Per-version fat jars produced from line-parameterized single modules: 9 SQL
source connectors (\{{flink-sql-connector-<X>-cdc${source.fatjar.suffix}}}) +
the 4 pipeline connectors whose external dependency lines fork across
generations (kafka / paimon / elasticsearch / iceberg,
\{{flink-cdc-pipeline-connector-<X>${pipeline.fatjar.suffix}}}), plus the 2.x
source e2e CI job. Both fat-jar suffixes and the dist coordinates derive from
the single root \{{flink.line}} property, so a same-generation CI job selects
all artifacts of its line with one override; adding a future Flink line
requires no new connector or dist modules at all|
|6|Version-sensitive surface convergence: main-code reflection into
\{{org.apache.flink}} internals eliminated (legacy \{{SinkFunction}} /
\{{SourceFunction}} call sites routed through Adapters; legacy Adapter
signatures aligned to minimal common interfaces) + a bytecode-scan guard +
bilingual documentation refresh|
h2. 4. Key evidence
h3. 4.1 Cross-generation bytecode equivalence of the intermediate modules (the
basis for deploy-once)
Method: both generations fully built with the same JDK and Maven; thin jars
compared entry by entry (CRC32 + size); divergent classes dissected through
their javap constant pools.
* Three difference families were identified: (1) *annotation retention* - Flink
annotation \{{@Retention}} differs across generations and javac compiles the
retention into consumer classes (constant 2-byte delta, zero runtime impact;
cannot be removed and need not be); (2) the legacy subpackage migration; (3)
divergent signatures of the legacy Adapters. Families (2) and (3) have been
converged through Adapters.
* Post-convergence state: runtime 3/314 classes differ (all family 1); cli,
python and composer are byte-identical (the Yarn dist-jar lookup derives the
Flink line from the running jar's identity at runtime, so composer carries no
generation-stamped resource at all - see the phase-4 row).
* Conclusion: intermediate modules deployed from the flink1 line are fully
usable by flink2 downstream (all Flink dependencies are \{{provided}} and
non-transitive). The release flow in section 5 is grounded in this evidence.
h3. 4.2 The Adapter signature contract
Auditing all 17 Adapter pairs (at the compiled-artifact level, via javap)
produced this rule:
* *Contract:* the two generations' Adapters must have fully identical
self-declared member signatures (same FQCN, same parameters, same returns).
* *Boundary:* differences on the inheritance chain (parent FQCNs decided by
Flink's own API, e.g. 2.x promoting \{{StatefulSink$StatefulSinkWriter}} to a
top-level interface) are the bridge class's job and are digested by the
per-line fat jars - they do not violate the contract.
* *Enforcement:* new direct references from main code to version-sensitive
Flink APIs are caught by the bytecode-scan guard and the review checklist.
h3. 4.3 Artifact equivalence certification
* Phase-5 acceptance: 22 key jars compared entry by entry against the
pre-refactor baseline (fat jars differ only by +GAV metadata / -old metadata;
class content unchanged).
* Pipeline connectors: the per-version fat jars are class-level identical to
the pre-refactor artifacts of their corresponding lines (the new-GAV pom
entries aside; the SQL source and dist fat jars are byte-identical including
their OSS placeholder class, and the 4 pipeline fat jars each drop exactly the
placeholder class the old per-line shells used to contribute - every real class
matches).
* The 4-fork vs 11-non-fork split was determined by a full jar-level scan of
all 15 pipeline connectors (e.g. the fluss fat jar is cross-generation
identical - its shaded \{{fluss-client:1.0.0}} is version-pinned and the
generation-specific properties are used only in tests).
h3. 4.4 Reflection elimination
All main-code reflection into \{{org.apache.flink}} internals has been
converged into the compatibility layer (constructors, fields, setup calls,
\{{Class.forName}} probes), enforced by a guard test that scans bytecode
constant-pool signatures (the violation whitelist is empty). Reflection that
instantiates UDFs by user-supplied class name is exempt - its target is user
code, not Flink.
h2. 5. Release process (finalized; community confirmation requested)
Two invocations of \{{deploy_staging_jars.sh}}, one per Flink line:
# *flink1 line* (\{{FLINK_VERSION=1.20.3}}): full deploy. The intermediate
modules enter staging exactly once here (their content is cross-generation
equivalent, see 4.1), together with all \{{-flink-1.20}} artifacts and the 11
single-GAV pipeline connector fat jars.
# *flink2 line* (\{{FLINK_VERSION=2.2.0}}): a full \{{install}} first
(compile/verification as usual, populates the local repository), then a
\{{-pl}}-restricted deploy of the per-version artifacts
(\{{flink-cdc-dist-flink-2.2}} + 9 \{{flink-sql-connector-*-flink-2.2}} + 4
\{{flink-cdc-pipeline-connector-*-flink-2.2}}). Deliberately *without*
\{{-am}}: otherwise the shared-GAV upstream modules would be deployed a second
time and rejected by Nexus staging (same GAV, different checksum).
A side benefit: the flink2 line's \{{-pl}} restriction naturally keeps the
starrocks/hudi flink2 thin jars out of staging. During a transition period the
release also emits legacy-named files (e.g. \{{flink-cdc-dist-3.8.0-1.20.jar}})
so download links stay alive; by default for one minor cycle.
h2. 6. How the change will be submitted
Everything is submitted as a *single PR* containing the six phase commits above:
* Reviewers can review commit by commit - each commit is an independent,
behavior-preserving change that can be reverted on its own without entangling
the rest (the phases share only build ordering, no revert coupling).
* The PR description will carry per-commit verification summaries and link back
to this ticket.
* If the reviewers prefer multiple PRs, the chained merge order is simply the
commit order (1 -> 6).
h2. 7. Decision points (feedback most valuable here)
||#||Topic||Implemented as||Alternative and its costs||
|D1|Artifact naming|artifactId suffix + clean version string (the Paimon/Fluss
shape); the suffix is carried by line-parameterized single modules, so no
per-line module clones exist anywhere in the reactor|Keep the old
version-suffix convention (\{{3.8.0-1.20}}): costs one extra full release build
per added Flink version, makes single-build multi-artifact impossible for
same-generation minors (e.g. 2.2 + 2.3), and requires migrating all
coordinates, docs and download pages back. Per-line module clones (the
Paimon/Fluss mechanism) were also considered and rejected: they duplicate the
shade configuration per line and make every added line cost 15+ new modules,
while the parameterized form keeps the artifact layout identical with zero
module growth|
|D2|Release strategy|Two-line deploy: flink1 full deploy once + flink2
restricted to a per-version \{{-pl}} list; intermediate modules deployed
exactly once|There is no substantive alternative - a second full deploy would
be rejected by Nexus staging, and the rejection is protective (a flink1-line
pipeline fat jar would be unusable on a 2.x cluster). Feedback is requested on
the acceptability of the flow itself|
|D3|Guava shade|CDC-owned package name, no version prefix|Synergy with
FLINK-35071; the cost is roughly +3 MB per fat jar|
h2. 8. Settled positions (for awareness, not up for vote)
* *starrocks / hudi:* no fat jars are produced under the flink2 build (upstream
has no 2.x external connector artifacts). Build behavior: the modules still
compile and run their tests under \{{-Pflink2}}; their shade step is skipped
via \{{connector.generation.skip}}, so only the thin jar is installed; the
flink2-line deploy \{{-pl}} list keeps those thin jars out of staging, so
Central only ever carries the flink1-line fat jars. CI and README state "1.20
only".
* *doris:* stays on the 1.20-line external dependency (the upstream flink2-line
connector is Java 17 bytecode, conflicting with the JDK 11 baseline). Build
behavior: the module compiles, tests and packages a full fat jar under both
profiles, but the dependency coordinates pin the external SDK to the 1.20 line
(\{{flink-doris-connector-1.20}}), so the flink2-line fat jar is
content-equivalent to the flink1-line one and is likewise kept out of the
flink2 deploy list; docs are marked "not verified on Flink 2.x".
* *Adapter narrow-read:* Adapter parameters take the minimal common interface
(e.g. \{{RuntimeContext}}, not \{{StreamingRuntimeContext}}); the type a call
site happens to hold must not leak into the signature.
* *Single source tree:* no \{{src/main/flink1}}-style versioned source
directories.
h2. 9. Verification matrix (the acceptance bar for the PR)
Dual-generation full builds + full UTs + repository gates (spotless /
checkstyle) as the baseline; representative e2e runs on CI (two jobs added:
\{{source_e2e_2_x}}, \{{shade_guards_2_x}}); artifact equivalence certified
through jar-level comparison. Pre-existing environment-only failures unrelated
to this refactor are enumerated in the PR description.
h2. 10. Known leftovers (disclosed as they are)
* The tutorial pages (mysql-to-kafka, migrate-paimon-cdc-action, etc.) still
carry download-link tables in the old naming scheme and need a full-page
refresh (within the Phase 6 docs scope or a separate docs ticket).
----
AI-assisted tooling was used during implementation and documentation; in line
with the ASF Generative Tooling Guidance this will be disclosed via the
Generated-by trailer in the PR.
was:
h1. Multi-Flink-version compatibility refactor
h2. 1. Background
h3. 1.1 The support matrix keeps shrinking
Flink CDC's Flink support matrix has been contracting: 3.1 supported 4 Flink
versions (1.16-1.19), from 3.5 only 1.19/1.20 remained, and 3.6/3.7 support
just 1.20 + 2.2. The marginal cost of adding one Flink version is "a full round
of build configuration plus profile surgery", and that cost curve directly
determines how wide the matrix can be.
h3. 1.2 Four structural problems with the current mechanism
The current mechanism (3.7 / 3.8-SNAPSHOT) is "single source tree + whole-tree
Maven profile switching + dual-FQCN shadow modules":
* *Profile fragmentation.* Generation switching (1.20 vs 2.2) is scattered
across the local profiles of 7 modules; the root pom's \{{flink2}} profile does
not even switch \{{flink.version}}; the runtime module needs a
property-activated \{{*:*}} exclusion profile to neutralize the default one;
test classpaths are hand-assembled via a \{{copy-flink2-guava}} hack.
* *The guava package-name trap.* 89+ files hardcode \{{import
org.apache.flink.shaded.guava31.*}}; the \{{flink-shaded-guava}} package prefix
evolves with Flink (1.20 -> guava31, 2.2 -> guava33). Under the flink2 build,
main modules compile against 31.1 while dist packages 33.4 (verified via
\{{dependency:tree}}), and at runtime we are saved only by the guava31
compatibility bundle retained in the Flink 2.2 distribution (FLINK-39288 /
flink-cdc PR #4339) - an implicit dependency with no contract behind it.
* *Shadow-class linkage risk without per-version verification.* The dist and
SQL connector fat jars contain 30+ \{{org.apache.flink.*}} shadow classes
(backports / copies). Linking correctness depends entirely on the shadowed
classes happening to be binary-compatible with the real Flink APIs, and there
is no systematic verification.
* *Main-code reflection into Flink internals.* \{{DataSinkWriterOperator}}
reflectively probes the 3-arg \{{SinkWriterOperator}} constructor and falls
back to the 4-arg one; the composer reflectively reads the private
\{{StreamExecutionEnvironment.configuration}} field. These are silent breakage
points on every Flink upgrade.
Two adjacent gaps: the external connector dependency matrix is incomplete (the
doris / starrocks / hudi external connectors are hardcoded to the 1.20 line,
which is "fake compatibility" under flink2 builds without an explicit
declaration), and source-connector e2e coverage is a single 1.20 CI job.
h3. 1.3 Prior art
Paimon and Fluss have converged on the same pattern: *common code compiled
against a per-generation baseline + one thin module per supported Flink minor +
one artifact jar per version*, and neither depends on \{{flink-shaded-guava}}
(both relocate guava into their own package space).
They differ in how the common layer is compiled: Paimon compiles it per
generation (\{{paimon-flink1-common}} / \{{paimon-flink2-common}}), while Fluss
compiles everything once against a single baseline and routes all differences
through Adapter overrides plus backports. This proposal follows the Paimon
shape: flink-cdc's legacy \{{SourceFunction}} ecosystem (the source connectors,
\{{DebeziumSourceFunction}}, the legacy branches of 8 TableSources) and the
2.x-new APIs (\{{With*Topology}}, \{{StateRecoveryOptions}}) present a
bidirectional sensitivity surface far larger than Fluss's (a young project with
no legacy); a single baseline would maximize shadow-class linkage risk, whereas
per-generation compilation links every line's shadowed classes against that
line's real Flink API.
h2. 2. Proposal: the target architecture
h3. 2.1 Three-layer compatibility structure (following Paimon)
{noformat}
flink-cdc-flink/ # Flink version compatibility layer
├── flink-cdc-flink1-common/ # generation-common layer (compiled
against 1.20.3):
│ # 2.x API backports + 1.20-line
Adapters
├── flink-cdc-flink2-common/ # generation-common layer (compiled
against 2.2.0):
│ # copies of APIs removed in 2.x +
2.2-line Adapters
├── flink-cdc-flink-1.20/ # version thin module (skeleton +
exact-version UTs;
│ # will host intra-generation minor
differences)
└── flink-cdc-flink-2.2/
{noformat}
* *The root pom is the only generation switch point.* The \{{flink1}} (default)
and \{{flink2}} profiles centralize all generation-scoped properties; modules
keep zero local profiles (the 7 modules' fragmented profiles are gone).
* *Version-sensitive APIs must go through Adapters.* Main code may not directly
reference Flink APIs that fork across generations. The two generations'
Adapters share FQCN and have fully identical self-declared member signatures
(contract in 4.2); the forks are digested inside the compatibility layer.
Enforced by a bytecode-scan guard (\{{ReflectionGuardTest}}) plus a review
checklist.
* *Guava self-governance.* A \{{flink-cdc-shaded-guava}} module relocates guava
to \{{org.apache.flink.cdc.shaded.guava.*}} (no version prefix, so the
guava31->33 prefix drift cannot repeat), the root pom switches to it globally,
and fat jars ship their own copy. This proceeds in synergy with FLINK-35071.
h3. 2.2 Artifact layout
||Artifact class||GAV shape||Basis||
|Intermediate modules (common / runtime / composer / cli / python / debezium /
source-connector code modules)|Single GAV (e.g. \{{flink-cdc-runtime:3.8.0}}),
deployed exactly once|Cross-generation jar-level comparison: bytecode
differences reduced to Flink annotation \{{@Retention}} differences (2 bytes,
zero runtime impact) - see 4.1|
|Pipeline distribution|\{{flink-cdc-dist-flink-1.20}} /
\{{flink-cdc-dist-flink-2.2}}|Shaded content is strongly generation-bound|
|9 SQL source-connector fat jars|\{{flink-sql-connector-mysql-cdc-flink-1.20}}
/ \{{-flink-2.2}}, etc.|Same|
|4 pipeline-connector fat jars (kafka, paimon, elasticsearch,
iceberg)|\{{flink-cdc-pipeline-connector-kafka-flink-1.20}} / \{{-flink-2.2}},
etc.|The shaded external connector lines fork across generations (kafka fat
jar: 2918 vs 4521 entries)|
|The other 11 pipeline-connector fat jars|Single GAV kept|Jar-level comparison
proves cross-generation content equivalence (annotation/metadata-only
differences, or byte-identical)|
*Invariant:* the repository no longer contains any artifact whose content
varies across Flink generations while its GAV carries no generation identity.
The Flink line is carried by the artifactId suffix and the version string stays
clean (\{{3.8.0}}) - consistent with Paimon (\{{paimon-flink-1.20:1.4.2}}) and
Fluss.
h3. 2.3 Compatibility promises
* \{{@Public}} APIs are untouched.
* DataStream API users' dependency coordinates do not change (the thin
single-GAV artifacts are usable on both generations - cross-generation
equivalence is verified).
* Zero runtime behavior change: every fat jar was compared class-by-class (name
+ CRC32) against the pre-refactor artifact of the corresponding Flink line;
class content is identical (a placeholder class aside).
* JDK 11 baseline, Maven 3.8.6+, single reactor - unchanged.
h2. 3. Implementation status
Implementation is complete on a feature branch: six phases, one commit each -
complete, independently reviewable change units. This is not a request for
permission to proceed, but a request to review the direction and the concrete
shape.
||Phase||Scope of the commit||
|1|Guava self-governance: a \{{flink-cdc-shaded-guava}} module relocating guava
to \{{org.apache.flink.cdc.shaded.guava.*}} (no version prefix) + a one-shot
import replacement across 89+ files + NOTICE/LICENSE updates|
|2|Build hygiene: the root pom becomes the single generation switch point; the
7 modules' fragmented local profiles, the neutralizing profiles and the
\{{copy-flink2-guava}} classpath hack are removed; starrocks/hudi flink2 output
gating with CI and README declarations|
|3|Compatibility layer reorg: the \{{flink-cdc-flink/}} aggregate with the two
generation-common modules (migrated and renamed, zero class-content change) and
the two version thin modules (skeleton + exact-version UTs), each thin module
pinned to its line-scoped Flink version so a future generation-baseline bump
cannot drag an older line along|
|4|Per-version dist produced from a single line-parameterized module:
\{{flink-cdc-dist}} shades the pipeline core and assembles the bin tarball, and
its artifactId carries the Flink line of the build
(\{{flink-cdc-dist-flink-${flink.line}}}), so every supported line gets its
distribution from one module with no per-line clones. The Yarn dist-jar lookup
derives the Flink line at runtime from the running jar's own identity
(\{{flink-cdc-dist-flink-<ver>-*.jar}} name parse) instead of a build-time
filtered resource - a single build-time value cannot serve two same-generation
releases (e.g. a future 2.2 + 2.3 produced by one \{{-Pflink2}} build), while
self-identifying jars always answer for themselves; unidentifiable classpath
shapes fail loudly. Release-script adaptation + transition-period legacy-named
artifacts; the dist shade-priority guard ITCase lives in the module and skips
itself on lines without the marker|
|5|Per-version fat jars produced from line-parameterized single modules: 9 SQL
source connectors (\{{flink-sql-connector-<X>-cdc${source.fatjar.suffix}}}) +
the 4 pipeline connectors whose external dependency lines fork across
generations (kafka / paimon / elasticsearch / iceberg,
\{{flink-cdc-pipeline-connector-<X>${pipeline.fatjar.suffix}}}), plus the 2.x
source e2e CI job. Both fat-jar suffixes and the dist coordinates derive from
the single root \{{flink.line}} property, so a same-generation CI job selects
all artifacts of its line with one override; adding a future Flink line
requires no new connector or dist modules at all|
|6|Version-sensitive surface convergence: main-code reflection into
\{{org.apache.flink}} internals eliminated (legacy \{{SinkFunction}} /
\{{SourceFunction}} call sites routed through Adapters; legacy Adapter
signatures aligned to minimal common interfaces) + a bytecode-scan guard +
bilingual documentation refresh|
h2. 4. Key evidence
h3. 4.1 Cross-generation bytecode equivalence of the intermediate modules (the
basis for deploy-once)
Method: both generations fully built with the same JDK and Maven; thin jars
compared entry by entry (CRC32 + size); divergent classes dissected through
their javap constant pools.
* Three difference families were identified: (1) *annotation retention* - Flink
annotation \{{@Retention}} differs across generations and javac compiles the
retention into consumer classes (constant 2-byte delta, zero runtime impact;
cannot be removed and need not be); (2) the legacy subpackage migration; (3)
divergent signatures of the legacy Adapters. Families (2) and (3) have been
converged through Adapters.
* Post-convergence state: runtime 3/314 classes differ (all family 1); cli,
python and composer are byte-identical (the Yarn dist-jar lookup derives the
Flink line from the running jar's identity at runtime, so composer carries no
generation-stamped resource at all - see the phase-4 row).
* Conclusion: intermediate modules deployed from the flink1 line are fully
usable by flink2 downstream (all Flink dependencies are \{{provided}} and
non-transitive). The release flow in section 5 is grounded in this evidence.
h3. 4.2 The Adapter signature contract
Auditing all 17 Adapter pairs (at the compiled-artifact level, via javap)
produced this rule:
* *Contract:* the two generations' Adapters must have fully identical
self-declared member signatures (same FQCN, same parameters, same returns).
* *Boundary:* differences on the inheritance chain (parent FQCNs decided by
Flink's own API, e.g. 2.x promoting \{{StatefulSink$StatefulSinkWriter}} to a
top-level interface) are the bridge class's job and are digested by the
per-line fat jars - they do not violate the contract.
* *Enforcement:* new direct references from main code to version-sensitive
Flink APIs are caught by the bytecode-scan guard and the review checklist.
h3. 4.3 Artifact equivalence certification
* Phase-5 acceptance: 22 key jars compared entry by entry against the
pre-refactor baseline (fat jars differ only by +GAV metadata / -old metadata;
class content unchanged).
* Pipeline connectors: the per-version fat jars are class-level identical to
the pre-refactor artifacts of their corresponding lines (the new-GAV pom
entries aside; the SQL source and dist fat jars are byte-identical including
their OSS placeholder class, and the 4 pipeline fat jars each drop exactly the
placeholder class the old per-line shells used to contribute - every real class
matches).
* The 4-fork vs 11-non-fork split was determined by a full jar-level scan of
all 15 pipeline connectors (e.g. the fluss fat jar is cross-generation
identical - its shaded \{{fluss-client:1.0.0}} is version-pinned and the
generation-specific properties are used only in tests).
h3. 4.4 Reflection elimination
All main-code reflection into \{{org.apache.flink}} internals has been
converged into the compatibility layer (constructors, fields, setup calls,
\{{Class.forName}} probes), enforced by a guard test that scans bytecode
constant-pool signatures (the violation whitelist is empty). Reflection that
instantiates UDFs by user-supplied class name is exempt - its target is user
code, not Flink.
h2. 5. Release process (finalized; community confirmation requested)
Two invocations of \{{deploy_staging_jars.sh}}, one per Flink line:
# *flink1 line* (\{{FLINK_VERSION=1.20.3}}): full deploy. The intermediate
modules enter staging exactly once here (their content is cross-generation
equivalent, see 4.1), together with all \{{-flink-1.20}} artifacts and the 11
single-GAV pipeline connector fat jars.
# *flink2 line* (\{{FLINK_VERSION=2.2.0}}): a full \{{install}} first
(compile/verification as usual, populates the local repository), then a
\{{-pl}}-restricted deploy of the per-version artifacts
(\{{flink-cdc-dist-flink-2.2}} + 9 \{{flink-sql-connector-*-flink-2.2}} + 4
\{{flink-cdc-pipeline-connector-*-flink-2.2}}). Deliberately *without*
\{{-am}}: otherwise the shared-GAV upstream modules would be deployed a second
time and rejected by Nexus staging (same GAV, different checksum).
A side benefit: the flink2 line's \{{-pl}} restriction naturally keeps the
starrocks/hudi flink2 thin jars out of staging. During a transition period the
release also emits legacy-named files (e.g. \{{flink-cdc-dist-3.8.0-1.20.jar}})
so download links stay alive; by default for one minor cycle.
h2. 6. How the change will be submitted
Everything is submitted as a *single PR* containing the six phase commits above:
* Reviewers can review commit by commit - each commit is an independent,
behavior-preserving change that can be reverted on its own without entangling
the rest (the phases share only build ordering, no revert coupling).
* The PR description will carry per-commit verification summaries and link back
to this ticket.
* If the reviewers prefer multiple PRs, the chained merge order is simply the
commit order (1 -> 6).
h2. 7. Decision points (feedback most valuable here)
||#||Topic||Implemented as||Alternative and its costs||
|D1|Artifact naming|artifactId suffix + clean version string (the Paimon/Fluss
shape); the suffix is carried by line-parameterized single modules, so no
per-line module clones exist anywhere in the reactor|Keep the old
version-suffix convention (\{{3.8.0-1.20}}): costs one extra full release build
per added Flink version, makes single-build multi-artifact impossible for
same-generation minors (e.g. 2.2 + 2.3), and requires migrating all
coordinates, docs and download pages back. Per-line module clones (the
Paimon/Fluss mechanism) were also considered and rejected: they duplicate the
shade configuration per line and make every added line cost 15+ new modules,
while the parameterized form keeps the artifact layout identical with zero
module growth|
|D2|Release strategy|Two-line deploy: flink1 full deploy once + flink2
restricted to a per-version \{{-pl}} list; intermediate modules deployed
exactly once|There is no substantive alternative - a second full deploy would
be rejected by Nexus staging, and the rejection is protective (a flink1-line
pipeline fat jar would be unusable on a 2.x cluster). Feedback is requested on
the acceptability of the flow itself|
|D3|Guava shade|CDC-owned package name, no version prefix|Synergy with
FLINK-35071; the cost is roughly +3 MB per fat jar|
h2. 8. Settled positions (for awareness, not up for vote)
* *starrocks / hudi:* no fat jars are produced under the flink2 build (upstream
has no 2.x external connector artifacts); CI and README state "1.20 only".
* *doris:* stays on the 1.20-line external dependency (the upstream flink2-line
connector is Java 17 bytecode, conflicting with the JDK 11 baseline); it still
compiles, tests and packages as before, with docs marked "not verified on Flink
2.x".
* *Adapter narrow-read:* Adapter parameters take the minimal common interface
(e.g. \{{RuntimeContext}}, not \{{StreamingRuntimeContext}}); the type a call
site happens to hold must not leak into the signature.
* *Single source tree:* no \{{src/main/flink1}}-style versioned source
directories.
h2. 9. Verification matrix (the acceptance bar for the PR)
Dual-generation full builds + full UTs + repository gates (spotless /
checkstyle) as the baseline; representative e2e runs on CI (two jobs added:
\{{source_e2e_2_x}}, \{{shade_guards_2_x}}); artifact equivalence certified
through jar-level comparison. Pre-existing environment-only failures unrelated
to this refactor are enumerated in the PR description.
h2. 10. Known leftovers (disclosed as they are)
* The tutorial pages (mysql-to-kafka, migrate-paimon-cdc-action, etc.) still
carry download-link tables in the old naming scheme and need a full-page
refresh (within the Phase 6 docs scope or a separate docs ticket).
----
AI-assisted tooling was used during implementation and documentation; in line
with the ASF Generative Tooling Guidance this will be disclosed via the
Generated-by trailer in the PR.
> Multi-Flink-version compatibility refactor
> ------------------------------------------
>
> Key: FLINK-41452
> URL: https://issues.apache.org/jira/browse/FLINK-41452
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.7.0
> Reporter: peiyu
> Priority: Major
> Labels: pull-request-available
>
> h1. Multi-Flink-version compatibility refactor
> h2. 1. Background
> h3. 1.1 The support matrix keeps shrinking
> Flink CDC's Flink support matrix has been contracting: 3.1 supported 4 Flink
> versions (1.16-1.19), from 3.5 only 1.19/1.20 remained, and 3.6/3.7 support
> just 1.20 + 2.2. The marginal cost of adding one Flink version is "a full
> round of build configuration plus profile surgery", and that cost curve
> directly determines how wide the matrix can be.
> h3. 1.2 Four structural problems with the current mechanism
> The current mechanism (3.7 / 3.8-SNAPSHOT) is "single source tree +
> whole-tree Maven profile switching + dual-FQCN shadow modules":
> * *Profile fragmentation.* Generation switching (1.20 vs 2.2) is scattered
> across the local profiles of 7 modules; the root pom's \{{flink2}} profile
> does not even switch \{{flink.version}}; the runtime module needs a
> property-activated \{{*:*}} exclusion profile to neutralize the default one;
> test classpaths are hand-assembled via a \{{copy-flink2-guava}} hack.
> * *The guava package-name trap.* 89+ files hardcode \{{import
> org.apache.flink.shaded.guava31.*}}; the \{{flink-shaded-guava}} package
> prefix evolves with Flink (1.20 -> guava31, 2.2 -> guava33). Under the flink2
> build, main modules compile against 31.1 while dist packages 33.4 (verified
> via \{{dependency:tree}}), and at runtime we are saved only by the guava31
> compatibility bundle retained in the Flink 2.2 distribution (FLINK-39288 /
> flink-cdc PR #4339) - an implicit dependency with no contract behind it.
> * *Shadow-class linkage risk without per-version verification.* The dist and
> SQL connector fat jars contain 30+ \{{org.apache.flink.*}} shadow classes
> (backports / copies). Linking correctness depends entirely on the shadowed
> classes happening to be binary-compatible with the real Flink APIs, and there
> is no systematic verification.
> * *Main-code reflection into Flink internals.* \{{DataSinkWriterOperator}}
> reflectively probes the 3-arg \{{SinkWriterOperator}} constructor and falls
> back to the 4-arg one; the composer reflectively reads the private
> \{{StreamExecutionEnvironment.configuration}} field. These are silent
> breakage points on every Flink upgrade.
> Two adjacent gaps: the external connector dependency matrix is incomplete
> (the doris / starrocks / hudi external connectors are hardcoded to the 1.20
> line, which is "fake compatibility" under flink2 builds without an explicit
> declaration), and source-connector e2e coverage is a single 1.20 CI job.
> h3. 1.3 Prior art
> Paimon and Fluss have converged on the same pattern: *common code compiled
> against a per-generation baseline + one thin module per supported Flink minor
> + one artifact jar per version*, and neither depends on
> \{{flink-shaded-guava}} (both relocate guava into their own package space).
> They differ in how the common layer is compiled: Paimon compiles it per
> generation (\{{paimon-flink1-common}} / \{{paimon-flink2-common}}), while
> Fluss compiles everything once against a single baseline and routes all
> differences through Adapter overrides plus backports. This proposal follows
> the Paimon shape: flink-cdc's legacy \{{SourceFunction}} ecosystem (the
> source connectors, \{{DebeziumSourceFunction}}, the legacy branches of 8
> TableSources) and the 2.x-new APIs (\{{With*Topology}},
> \{{StateRecoveryOptions}}) present a bidirectional sensitivity surface far
> larger than Fluss's (a young project with no legacy); a single baseline would
> maximize shadow-class linkage risk, whereas per-generation compilation links
> every line's shadowed classes against that line's real Flink API.
> h2. 2. Proposal: the target architecture
> h3. 2.1 Three-layer compatibility structure (following Paimon)
> {noformat}
> flink-cdc-flink/ # Flink version compatibility layer
> ├── flink-cdc-flink1-common/ # generation-common layer (compiled
> against 1.20.3):
> │ # 2.x API backports + 1.20-line
> Adapters
> ├── flink-cdc-flink2-common/ # generation-common layer (compiled
> against 2.2.0):
> │ # copies of APIs removed in 2.x +
> 2.2-line Adapters
> ├── flink-cdc-flink-1.20/ # version thin module (skeleton +
> exact-version UTs;
> │ # will host intra-generation minor
> differences)
> └── flink-cdc-flink-2.2/
> {noformat}
> * *The root pom is the only generation switch point.* The \{{flink1}}
> (default) and \{{flink2}} profiles centralize all generation-scoped
> properties; modules keep zero local profiles (the 7 modules' fragmented
> profiles are gone).
> * *Version-sensitive APIs must go through Adapters.* Main code may not
> directly reference Flink APIs that fork across generations. The two
> generations' Adapters share FQCN and have fully identical self-declared
> member signatures (contract in 4.2); the forks are digested inside the
> compatibility layer. Enforced by a bytecode-scan guard
> (\{{ReflectionGuardTest}}) plus a review checklist.
> * *Guava self-governance.* A \{{flink-cdc-shaded-guava}} module relocates
> guava to \{{org.apache.flink.cdc.shaded.guava.*}} (no version prefix, so the
> guava31->33 prefix drift cannot repeat), the root pom switches to it
> globally, and fat jars ship their own copy. This proceeds in synergy with
> FLINK-35071.
> h3. 2.2 Artifact layout
> ||Artifact class||GAV shape||Basis||
> |Intermediate modules (common / runtime / composer / cli / python / debezium
> / source-connector code modules)|Single GAV (e.g.
> \{{flink-cdc-runtime:3.8.0}}), deployed exactly once|Cross-generation
> jar-level comparison: bytecode differences reduced to Flink annotation
> \{{@Retention}} differences (2 bytes, zero runtime impact) - see 4.1|
> |Pipeline distribution|\{{flink-cdc-dist-flink-1.20}} /
> \{{flink-cdc-dist-flink-2.2}}|Shaded content is strongly generation-bound|
> |9 SQL source-connector fat
> jars|\{{flink-sql-connector-mysql-cdc-flink-1.20}} / \{{-flink-2.2}},
> etc.|Same|
> |4 pipeline-connector fat jars (kafka, paimon, elasticsearch,
> iceberg)|\{{flink-cdc-pipeline-connector-kafka-flink-1.20}} /
> \{{-flink-2.2}}, etc.|The shaded external connector lines fork across
> generations (kafka fat jar: 2918 vs 4521 entries)|
> |The other 11 pipeline-connector fat jars|Single GAV kept|Jar-level
> comparison proves cross-generation content equivalence
> (annotation/metadata-only differences, or byte-identical)|
> *Invariant:* the repository no longer contains any artifact whose content
> varies across Flink generations while its GAV carries no generation identity.
> The Flink line is carried by the artifactId suffix and the version string
> stays clean (\{{3.8.0}}) - consistent with Paimon
> (\{{paimon-flink-1.20:1.4.2}}) and Fluss.
> h3. 2.3 Compatibility promises
> * \{{@Public}} APIs are untouched.
> * DataStream API users' dependency coordinates do not change (the thin
> single-GAV artifacts are usable on both generations - cross-generation
> equivalence is verified).
> * Zero runtime behavior change: every fat jar was compared class-by-class
> (name + CRC32) against the pre-refactor artifact of the corresponding Flink
> line; class content is identical (a placeholder class aside).
> * JDK 11 baseline, Maven 3.8.6+, single reactor - unchanged.
> h2. 3. Implementation status
> Implementation is complete on a feature branch: six phases, one commit each -
> complete, independently reviewable change units. This is not a request for
> permission to proceed, but a request to review the direction and the concrete
> shape.
> ||Phase||Scope of the commit||
> |1|Guava self-governance: a \{{flink-cdc-shaded-guava}} module relocating
> guava to \{{org.apache.flink.cdc.shaded.guava.*}} (no version prefix) + a
> one-shot import replacement across 89+ files + NOTICE/LICENSE updates|
> |2|Build hygiene: the root pom becomes the single generation switch point;
> the 7 modules' fragmented local profiles, the neutralizing profiles and the
> \{{copy-flink2-guava}} classpath hack are removed; starrocks/hudi flink2
> output gating with CI and README declarations|
> |3|Compatibility layer reorg: the \{{flink-cdc-flink/}} aggregate with the
> two generation-common modules (migrated and renamed, zero class-content
> change) and the two version thin modules (skeleton + exact-version UTs), each
> thin module pinned to its line-scoped Flink version so a future
> generation-baseline bump cannot drag an older line along|
> |4|Per-version dist produced from a single line-parameterized module:
> \{{flink-cdc-dist}} shades the pipeline core and assembles the bin tarball,
> and its artifactId carries the Flink line of the build
> (\{{flink-cdc-dist-flink-${flink.line}}}), so every supported line gets its
> distribution from one module with no per-line clones. The Yarn dist-jar
> lookup derives the Flink line at runtime from the running jar's own identity
> (\{{flink-cdc-dist-flink-<ver>-*.jar}} name parse) instead of a build-time
> filtered resource - a single build-time value cannot serve two
> same-generation releases (e.g. a future 2.2 + 2.3 produced by one
> \{{-Pflink2}} build), while self-identifying jars always answer for
> themselves; unidentifiable classpath shapes fail loudly. Release-script
> adaptation + transition-period legacy-named artifacts; the dist
> shade-priority guard ITCase lives in the module and skips itself on lines
> without the marker|
> |5|Per-version fat jars produced from line-parameterized single modules: 9
> SQL source connectors
> (\{{flink-sql-connector-<X>-cdc${source.fatjar.suffix}}}) + the 4 pipeline
> connectors whose external dependency lines fork across generations (kafka /
> paimon / elasticsearch / iceberg,
> \{{flink-cdc-pipeline-connector-<X>${pipeline.fatjar.suffix}}}), plus the 2.x
> source e2e CI job. Both fat-jar suffixes and the dist coordinates derive from
> the single root \{{flink.line}} property, so a same-generation CI job selects
> all artifacts of its line with one override; adding a future Flink line
> requires no new connector or dist modules at all|
> |6|Version-sensitive surface convergence: main-code reflection into
> \{{org.apache.flink}} internals eliminated (legacy \{{SinkFunction}} /
> \{{SourceFunction}} call sites routed through Adapters; legacy Adapter
> signatures aligned to minimal common interfaces) + a bytecode-scan guard +
> bilingual documentation refresh|
> h2. 4. Key evidence
> h3. 4.1 Cross-generation bytecode equivalence of the intermediate modules
> (the basis for deploy-once)
> Method: both generations fully built with the same JDK and Maven; thin jars
> compared entry by entry (CRC32 + size); divergent classes dissected through
> their javap constant pools.
> * Three difference families were identified: (1) *annotation retention* -
> Flink annotation \{{@Retention}} differs across generations and javac
> compiles the retention into consumer classes (constant 2-byte delta, zero
> runtime impact; cannot be removed and need not be); (2) the legacy subpackage
> migration; (3) divergent signatures of the legacy Adapters. Families (2) and
> (3) have been converged through Adapters.
> * Post-convergence state: runtime 3/314 classes differ (all family 1); cli,
> python and composer are byte-identical (the Yarn dist-jar lookup derives the
> Flink line from the running jar's identity at runtime, so composer carries no
> generation-stamped resource at all - see the phase-4 row).
> * Conclusion: intermediate modules deployed from the flink1 line are fully
> usable by flink2 downstream (all Flink dependencies are \{{provided}} and
> non-transitive). The release flow in section 5 is grounded in this evidence.
> h3. 4.2 The Adapter signature contract
> Auditing all 17 Adapter pairs (at the compiled-artifact level, via javap)
> produced this rule:
> * *Contract:* the two generations' Adapters must have fully identical
> self-declared member signatures (same FQCN, same parameters, same returns).
> * *Boundary:* differences on the inheritance chain (parent FQCNs decided by
> Flink's own API, e.g. 2.x promoting \{{StatefulSink$StatefulSinkWriter}} to a
> top-level interface) are the bridge class's job and are digested by the
> per-line fat jars - they do not violate the contract.
> * *Enforcement:* new direct references from main code to version-sensitive
> Flink APIs are caught by the bytecode-scan guard and the review checklist.
> h3. 4.3 Artifact equivalence certification
> * Phase-5 acceptance: 22 key jars compared entry by entry against the
> pre-refactor baseline (fat jars differ only by +GAV metadata / -old metadata;
> class content unchanged).
> * Pipeline connectors: the per-version fat jars are class-level identical to
> the pre-refactor artifacts of their corresponding lines (the new-GAV pom
> entries aside; the SQL source and dist fat jars are byte-identical including
> their OSS placeholder class, and the 4 pipeline fat jars each drop exactly
> the placeholder class the old per-line shells used to contribute - every real
> class matches).
> * The 4-fork vs 11-non-fork split was determined by a full jar-level scan of
> all 15 pipeline connectors (e.g. the fluss fat jar is cross-generation
> identical - its shaded \{{fluss-client:1.0.0}} is version-pinned and the
> generation-specific properties are used only in tests).
> h3. 4.4 Reflection elimination
> All main-code reflection into \{{org.apache.flink}} internals has been
> converged into the compatibility layer (constructors, fields, setup calls,
> \{{Class.forName}} probes), enforced by a guard test that scans bytecode
> constant-pool signatures (the violation whitelist is empty). Reflection that
> instantiates UDFs by user-supplied class name is exempt - its target is user
> code, not Flink.
> h2. 5. Release process (finalized; community confirmation requested)
> Two invocations of \{{deploy_staging_jars.sh}}, one per Flink line:
> # *flink1 line* (\{{FLINK_VERSION=1.20.3}}): full deploy. The intermediate
> modules enter staging exactly once here (their content is cross-generation
> equivalent, see 4.1), together with all \{{-flink-1.20}} artifacts and the 11
> single-GAV pipeline connector fat jars.
> # *flink2 line* (\{{FLINK_VERSION=2.2.0}}): a full \{{install}} first
> (compile/verification as usual, populates the local repository), then a
> \{{-pl}}-restricted deploy of the per-version artifacts
> (\{{flink-cdc-dist-flink-2.2}} + 9 \{{flink-sql-connector-*-flink-2.2}} + 4
> \{{flink-cdc-pipeline-connector-*-flink-2.2}}). Deliberately *without*
> \{{-am}}: otherwise the shared-GAV upstream modules would be deployed a
> second time and rejected by Nexus staging (same GAV, different checksum).
> A side benefit: the flink2 line's \{{-pl}} restriction naturally keeps the
> starrocks/hudi flink2 thin jars out of staging. During a transition period
> the release also emits legacy-named files (e.g.
> \{{flink-cdc-dist-3.8.0-1.20.jar}}) so download links stay alive; by default
> for one minor cycle.
> h2. 6. How the change will be submitted
> Everything is submitted as a *single PR* containing the six phase commits
> above:
> * Reviewers can review commit by commit - each commit is an independent,
> behavior-preserving change that can be reverted on its own without entangling
> the rest (the phases share only build ordering, no revert coupling).
> * The PR description will carry per-commit verification summaries and link
> back to this ticket.
> * If the reviewers prefer multiple PRs, the chained merge order is simply the
> commit order (1 -> 6).
> h2. 7. Decision points (feedback most valuable here)
> ||#||Topic||Implemented as||Alternative and its costs||
> |D1|Artifact naming|artifactId suffix + clean version string (the
> Paimon/Fluss shape); the suffix is carried by line-parameterized single
> modules, so no per-line module clones exist anywhere in the reactor|Keep the
> old version-suffix convention (\{{3.8.0-1.20}}): costs one extra full release
> build per added Flink version, makes single-build multi-artifact impossible
> for same-generation minors (e.g. 2.2 + 2.3), and requires migrating all
> coordinates, docs and download pages back. Per-line module clones (the
> Paimon/Fluss mechanism) were also considered and rejected: they duplicate the
> shade configuration per line and make every added line cost 15+ new modules,
> while the parameterized form keeps the artifact layout identical with zero
> module growth|
> |D2|Release strategy|Two-line deploy: flink1 full deploy once + flink2
> restricted to a per-version \{{-pl}} list; intermediate modules deployed
> exactly once|There is no substantive alternative - a second full deploy would
> be rejected by Nexus staging, and the rejection is protective (a flink1-line
> pipeline fat jar would be unusable on a 2.x cluster). Feedback is requested
> on the acceptability of the flow itself|
> |D3|Guava shade|CDC-owned package name, no version prefix|Synergy with
> FLINK-35071; the cost is roughly +3 MB per fat jar|
> h2. 8. Settled positions (for awareness, not up for vote)
> * *starrocks / hudi:* no fat jars are produced under the flink2 build
> (upstream has no 2.x external connector artifacts). Build behavior: the
> modules still compile and run their tests under \{{-Pflink2}}; their shade
> step is skipped via \{{connector.generation.skip}}, so only the thin jar is
> installed; the flink2-line deploy \{{-pl}} list keeps those thin jars out of
> staging, so Central only ever carries the flink1-line fat jars. CI and README
> state "1.20 only".
> * *doris:* stays on the 1.20-line external dependency (the upstream
> flink2-line connector is Java 17 bytecode, conflicting with the JDK 11
> baseline). Build behavior: the module compiles, tests and packages a full fat
> jar under both profiles, but the dependency coordinates pin the external SDK
> to the 1.20 line (\{{flink-doris-connector-1.20}}), so the flink2-line fat
> jar is content-equivalent to the flink1-line one and is likewise kept out of
> the flink2 deploy list; docs are marked "not verified on Flink 2.x".
> * *Adapter narrow-read:* Adapter parameters take the minimal common interface
> (e.g. \{{RuntimeContext}}, not \{{StreamingRuntimeContext}}); the type a call
> site happens to hold must not leak into the signature.
> * *Single source tree:* no \{{src/main/flink1}}-style versioned source
> directories.
> h2. 9. Verification matrix (the acceptance bar for the PR)
> Dual-generation full builds + full UTs + repository gates (spotless /
> checkstyle) as the baseline; representative e2e runs on CI (two jobs added:
> \{{source_e2e_2_x}}, \{{shade_guards_2_x}}); artifact equivalence certified
> through jar-level comparison. Pre-existing environment-only failures
> unrelated to this refactor are enumerated in the PR description.
> h2. 10. Known leftovers (disclosed as they are)
> * The tutorial pages (mysql-to-kafka, migrate-paimon-cdc-action, etc.) still
> carry download-link tables in the old naming scheme and need a full-page
> refresh (within the Phase 6 docs scope or a separate docs ticket).
> ----
> AI-assisted tooling was used during implementation and documentation; in line
> with the ASF Generative Tooling Guidance this will be disclosed via the
> Generated-by trailer in the PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
