[
https://issues.apache.org/jira/browse/FLINK-41452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated FLINK-41452:
-----------------------------------
Labels: pull-request-available (was: )
> Multi-Flink-version compatibility refactor
> ------------------------------------------
>
> Key: FLINK-41452
> URL: https://issues.apache.org/jira/browse/FLINK-41452
> Project: Flink
> Issue Type: Improvement
> Components: Flink CDC
> Affects Versions: cdc-3.7.0
> Reporter: peiyu
> Priority: Major
> Labels: pull-request-available
>
> h1. Multi-Flink-version compatibility refactor
> h2. 1. Background
> h3. 1.1 The support matrix keeps shrinking
> Flink CDC's Flink support matrix has been contracting: 3.1 supported 4 Flink
> versions (1.16-1.19), from 3.5 only 1.19/1.20 remained, and 3.6/3.7 support
> just 1.20 + 2.2. The marginal cost of adding one Flink version is "a full
> round of build configuration plus profile surgery", and that cost curve
> directly determines how wide the matrix can be.
> h3. 1.2 Four structural problems with the current mechanism
> The current mechanism (3.7 / 3.8-SNAPSHOT) is "single source tree +
> whole-tree Maven profile switching + dual-FQCN shadow modules":
> * *Profile fragmentation.* Generation switching (1.20 vs 2.2) is scattered
> across the local profiles of 7 modules; the root pom's {{flink2}} profile
> does not even switch {{flink.version}}; the runtime module needs a
> property-activated {{*:*}} exclusion profile to neutralize the default one;
> test classpaths are hand-assembled via a {{copy-flink2-guava}} hack.
> * *The guava package-name trap.* 89+ files hardcode {{import
> org.apache.flink.shaded.guava31.*}}; the {{flink-shaded-guava}} package
> prefix evolves with Flink (1.20 -> guava31, 2.2 -> guava33). Under the flink2
> build, main modules compile against 31.1 while dist packages 33.4 (verified
> via {{dependency:tree}}), and at runtime we are saved only by the guava31
> compatibility bundle retained in the Flink 2.2 distribution (FLINK-39288 /
> flink-cdc PR #4339) - an implicit dependency with no contract behind it.
> * *Shadow-class linkage risk without per-version verification.* The dist and
> SQL connector fat jars contain 30+ {{org.apache.flink.*}} shadow classes
> (backports / copies). Linking correctness depends entirely on the shadowed
> classes happening to be binary-compatible with the real Flink APIs, and there
> is no systematic verification.
> * *Main-code reflection into Flink internals.* {{DataSinkWriterOperator}}
> reflectively probes the 3-arg {{SinkWriterOperator}} constructor and falls
> back to the 4-arg one; the composer reflectively reads the private
> {{StreamExecutionEnvironment.configuration}} field. These are silent breakage
> points on every Flink upgrade.
> Two adjacent gaps: the external connector dependency matrix is incomplete
> (the doris / starrocks / hudi external connectors are hardcoded to the 1.20
> line, which is "fake compatibility" under flink2 builds without an explicit
> declaration), and source-connector e2e coverage is a single 1.20 CI job.
> h3. 1.3 Prior art
> Paimon and Fluss have converged on the same pattern: *common code compiled
> against a per-generation baseline + one thin module per supported Flink minor
> + one artifact jar per version*, and neither depends on
> {{flink-shaded-guava}} (both relocate guava into their own package space).
> They differ in how the common layer is compiled: Paimon compiles it per
> generation ({{paimon-flink1-common}} / {{paimon-flink2-common}}), while Fluss
> compiles everything once against a single baseline and routes all differences
> through Adapter overrides plus backports. This proposal follows the Paimon
> shape: flink-cdc's legacy {{SourceFunction}} ecosystem (the source
> connectors, {{DebeziumSourceFunction}}, the legacy branches of 8
> TableSources) and the 2.x-new APIs ({{With*Topology}},
> {{StateRecoveryOptions}}) present a bidirectional sensitivity surface far
> larger than Fluss's (a young project with no legacy); a single baseline would
> maximize shadow-class linkage risk, whereas per-generation compilation links
> every line's shadowed classes against that line's real Flink API.
> h2. 2. Proposal: the target architecture
> h3. 2.1 Three-layer compatibility structure (following Paimon)
> {noformat}
> flink-cdc-flink/ # Flink version compatibility layer
> ├── flink-cdc-flink1-common/ # generation-common layer (compiled
> against 1.20.3):
> │ # 2.x API backports + 1.20-line
> Adapters
> ├── flink-cdc-flink2-common/ # generation-common layer (compiled
> against 2.2.0):
> │ # copies of APIs removed in 2.x +
> 2.2-line Adapters
> ├── flink-cdc-flink-1.20/ # version thin module (skeleton +
> exact-version UTs;
> │ # will host intra-generation minor
> differences)
> └── flink-cdc-flink-2.2/
> {noformat}
> * *The root pom is the only generation switch point.* The {{flink1}}
> (default) and {{flink2}} profiles centralize all generation-scoped
> properties; modules keep zero local profiles (the 7 modules' fragmented
> profiles are gone).
> * *Version-sensitive APIs must go through Adapters.* Main code may not
> directly reference Flink APIs that fork across generations. The two
> generations' Adapters share FQCN and have fully identical self-declared
> member signatures (contract in 4.2); the forks are digested inside the
> compatibility layer. Enforced by a bytecode-scan guard
> ({{ReflectionGuardTest}}) plus a review checklist.
> * *Guava self-governance.* A {{flink-cdc-shaded-guava}} module relocates
> guava to {{org.apache.flink.cdc.shaded.guava.*}} (no version prefix, so the
> guava31->33 prefix drift cannot repeat), the root pom switches to it
> globally, and fat jars ship their own copy. This proceeds in synergy with
> FLINK-35071.
> h3. 2.2 Artifact layout
> ||Artifact class||GAV shape||Basis||
> |Intermediate modules (common / runtime / composer / cli / python / debezium
> / source-connector code modules)|Single GAV (e.g.
> {{flink-cdc-runtime:3.8.0}}), deployed exactly once|Cross-generation
> jar-level comparison: bytecode differences reduced to Flink annotation
> {{@Retention}} differences (2 bytes, zero runtime impact) - see 4.1|
> |Pipeline distribution|{{flink-cdc-dist-flink-1.20}} /
> {{flink-cdc-dist-flink-2.2}}|Shaded content is strongly generation-bound|
> |9 SQL source-connector fat jars|{{flink-sql-connector-mysql-cdc-flink-1.20}}
> / {{-flink-2.2}}, etc.|Same|
> |4 pipeline-connector fat jars (kafka, paimon, elasticsearch,
> iceberg)|{{flink-cdc-pipeline-connector-kafka-flink-1.20}} / {{-flink-2.2}},
> etc.|The shaded external connector lines fork across generations (kafka fat
> jar: 2918 vs 4521 entries)|
> |The other 11 pipeline-connector fat jars|Single GAV kept|Jar-level
> comparison proves cross-generation content equivalence
> (annotation/metadata-only differences, or byte-identical)|
> *Invariant:* the repository no longer contains any artifact whose content
> varies across Flink generations while its GAV carries no generation identity.
> The Flink line is carried by the artifactId suffix and the version string
> stays clean ({{3.8.0}}) - consistent with Paimon
> ({{paimon-flink-1.20:1.4.2}}) and Fluss.
> h3. 2.3 Compatibility promises
> * {{@Public}} APIs are untouched.
> * DataStream API users' dependency coordinates do not change (the thin
> single-GAV artifacts are usable on both generations - cross-generation
> equivalence is verified).
> * Zero runtime behavior change: every fat jar was compared class-by-class
> (name + CRC32) against the pre-refactor artifact of the corresponding Flink
> line; class content is identical (a placeholder class aside).
> * JDK 11 baseline, Maven 3.8.6+, single reactor - unchanged.
> h2. 3. Implementation status
> Implementation is complete on a feature branch: six phases, one commit each -
> complete, independently reviewable change units. This is not a request for
> permission to proceed, but a request to review the direction and the concrete
> shape.
> ||Phase||Scope of the commit||
> |1|Guava self-governance: a {{flink-cdc-shaded-guava}} module relocating
> guava to {{org.apache.flink.cdc.shaded.guava.*}} (no version prefix) + a
> one-shot import replacement across 89+ files + NOTICE/LICENSE updates|
> |2|Build hygiene: the root pom becomes the single generation switch point;
> the 7 modules' fragmented local profiles, the neutralizing profiles and the
> {{copy-flink2-guava}} classpath hack are removed; starrocks/hudi flink2
> output gating with CI and README declarations|
> |3|Compatibility layer reorg: the {{flink-cdc-flink/}} aggregate with the two
> generation-common modules (migrated and renamed, zero class-content change)
> and the two version thin modules (skeleton + exact-version UTs)|
> |4|Per-version dist: {{flink-cdc-dist-flink-1.20}} / {{-flink-2.2}} property
> gating + exact-match Yarn jar lookup + release-script adaptation +
> transition-period legacy-named artifacts|
> |5|Per-version fat jars: 9 SQL source connectors + the 4 pipeline connectors
> whose external dependency lines fork across generations (kafka / paimon /
> elasticsearch / iceberg) + the 2.x source e2e CI job|
> |6|Version-sensitive surface convergence: main-code reflection into
> {{org.apache.flink}} internals eliminated (legacy {{SinkFunction}} /
> {{SourceFunction}} call sites routed through Adapters; legacy Adapter
> signatures aligned to minimal common interfaces) + a bytecode-scan guard +
> bilingual documentation refresh|
> h2. 4. Key evidence
> h3. 4.1 Cross-generation bytecode equivalence of the intermediate modules
> (the basis for deploy-once)
> Method: both generations fully built with the same JDK and Maven; thin jars
> compared entry by entry (CRC32 + size); divergent classes dissected through
> their javap constant pools.
> * Three difference families were identified: (1) *annotation retention* -
> Flink annotation {{@Retention}} differs across generations and javac compiles
> the retention into consumer classes (constant 2-byte delta, zero runtime
> impact; cannot be removed and need not be); (2) the legacy subpackage
> migration; (3) divergent signatures of the legacy Adapters. Families (2) and
> (3) have been converged through Adapters.
> * Post-convergence state: runtime 3/314 classes differ (all family 1); cli
> and python are byte-identical; composer differs only in the
> {{flink-cdc-build.properties}} resource (consumed for Yarn-deploy jar lookup,
> and only inside the dist fat jars, which are per-version - it does not affect
> thin-jar cross-generation usability).
> * Conclusion: intermediate modules deployed from the flink1 line are fully
> usable by flink2 downstream (all Flink dependencies are {{provided}} and
> non-transitive). The release flow in section 5 is grounded in this evidence.
> h3. 4.2 The Adapter signature contract
> Auditing all 17 Adapter pairs (at the compiled-artifact level, via javap)
> produced this rule:
> * *Contract:* the two generations' Adapters must have fully identical
> self-declared member signatures (same FQCN, same parameters, same returns).
> * *Boundary:* differences on the inheritance chain (parent FQCNs decided by
> Flink's own API, e.g. 2.x promoting {{StatefulSink$StatefulSinkWriter}} to a
> top-level interface) are the bridge class's job and are digested by the
> per-line fat jars - they do not violate the contract.
> * *Enforcement:* new direct references from main code to version-sensitive
> Flink APIs are caught by the bytecode-scan guard and the review checklist.
> h3. 4.3 Artifact equivalence certification
> * Phase-5 acceptance: 22 key jars compared entry by entry against the
> pre-refactor baseline (fat jars differ only by +GAV metadata / -old metadata;
> class content unchanged).
> * Pipeline connectors: the 8 per-version fat jars are class-level identical
> to the pre-refactor artifacts of their corresponding lines (the new-GAV pom
> entries and a DummyDocs placeholder class aside).
> * The 4-fork vs 11-non-fork split was determined by a full jar-level scan of
> all 15 pipeline connectors (e.g. the fluss fat jar is cross-generation
> identical - its shaded {{fluss-client:1.0.0}} is version-pinned and the
> generation-specific properties are used only in tests).
> h3. 4.4 Reflection elimination
> All main-code reflection into {{org.apache.flink}} internals has been
> converged into the compatibility layer (constructors, fields, setup calls,
> {{Class.forName}} probes), enforced by a guard test that scans bytecode
> constant-pool signatures (the violation whitelist is empty). Reflection that
> instantiates UDFs by user-supplied class name is exempt - its target is user
> code, not Flink.
> h2. 5. Release process (finalized; community confirmation requested)
> Two invocations of {{deploy_staging_jars.sh}}, one per Flink line:
> # *flink1 line* ({{FLINK_VERSION=1.20.3}}): full deploy. The intermediate
> modules enter staging exactly once here (their content is cross-generation
> equivalent, see 4.1), together with all {{-flink-1.20}} artifacts and the 11
> single-GAV pipeline connector fat jars.
> # *flink2 line* ({{FLINK_VERSION=2.2.0}}): a full {{install}} first
> (compile/verification as usual, populates the local repository), then a
> {{-pl}}-restricted deploy of the per-version artifacts
> ({{flink-cdc-dist-flink-2.2}} + 9 {{flink-sql-connector-*-flink-2.2}} + 4
> {{flink-cdc-pipeline-connector-*-flink-2.2}}). Deliberately *without*
> {{-am}}: otherwise the shared-GAV upstream modules would be deployed a second
> time and rejected by Nexus staging (same GAV, different checksum).
> A side benefit: the flink2 line's {{-pl}} restriction naturally keeps the
> starrocks/hudi flink2 thin shells out of staging. During a transition period
> the release also emits legacy-named files (e.g.
> {{flink-cdc-dist-3.8.0-1.20.jar}}) so download links stay alive; by default
> for one minor cycle.
> h2. 6. How the change will be submitted
> Everything is submitted as a *single PR* containing the six phase commits
> above:
> * Reviewers can review commit by commit - each commit is an independent,
> behavior-preserving change that can be reverted on its own without entangling
> the rest (the phases share only build ordering, no revert coupling).
> * The PR description will carry per-commit verification summaries and link
> back to this ticket.
> * If the reviewers prefer multiple PRs, the chained merge order is simply the
> commit order (1 -> 6).
> h2. 7. Decision points (feedback most valuable here)
> ||#||Topic||Implemented as||Alternative and its costs||
> |D1|Artifact naming|artifactId suffix + clean version string (the
> Paimon/Fluss shape)|Keep the old version-suffix convention ({{3.8.0-1.20}}):
> costs one extra full release build per added Flink version, makes
> single-build multi-artifact impossible for same-generation minors (e.g. 2.2 +
> 2.3), and requires migrating all coordinates, docs and download pages back|
> |D2|Release strategy|Two-line deploy: flink1 full deploy once + flink2
> restricted to a per-version {{-pl}} list; intermediate modules deployed
> exactly once|There is no substantive alternative - a second full deploy would
> be rejected by Nexus staging, and the rejection is protective (a flink1-line
> pipeline fat jar would be unusable on a 2.x cluster). Feedback is requested
> on the acceptability of the flow itself|
> |D3|Guava shade|CDC-owned package name, no version prefix|Synergy with
> FLINK-35071; the cost is roughly +3 MB per fat jar|
> h2. 8. Settled positions (for awareness, not up for vote)
> * *starrocks / hudi:* no fat jars are produced under the flink2 build
> (upstream has no 2.x external connector artifacts); CI and README state "1.20
> only".
> * *doris:* stays on the 1.20-line external dependency (the upstream
> flink2-line connector is Java 17 bytecode, conflicting with the JDK 11
> baseline); it still compiles, tests and packages as before, with docs marked
> "not verified on Flink 2.x".
> * *Adapter narrow-read:* Adapter parameters take the minimal common interface
> (e.g. {{RuntimeContext}}, not {{StreamingRuntimeContext}}); the type a call
> site happens to hold must not leak into the signature.
> * *Single source tree:* no {{src/main/flink1}}-style versioned source
> directories.
> h2. 9. Verification matrix (the acceptance bar for the PR)
> Dual-generation full builds + full UTs + repository gates (spotless /
> checkstyle) as the baseline; representative e2e runs on CI (two jobs added:
> {{source_e2e_2_x}}, {{shade_guards_2_x}}); artifact equivalence certified
> through jar-level comparison. Pre-existing environment-only failures
> unrelated to this refactor are enumerated in the PR description.
> h2. 10. Known leftovers (disclosed as they are)
> * Runtime-ization of composer's {{flink-cdc-build.properties}} (optional
> hardening; Yarn jar lookup is already served correctly by the per-version
> dist jars).
> * The tutorial pages (mysql-to-kafka, migrate-paimon-cdc-action, etc.) still
> carry download-link tables in the old naming scheme and need a full-page
> refresh (within the Phase 6 docs scope or a separate docs ticket).
> * The shade configuration is duplicated across the per-version submodules of
> the 4 pipeline connectors (marked keep-in-sync) - the simplest viable shape
> under Maven's hard rules (a jar module cannot declare {{<modules>}}; a parent
> must be pom-packaged); it can be converged later if Maven relaxes or a
> plugin-level abstraction is introduced.
> ----
> AI-assisted tooling was used during implementation and documentation; in line
> with the ASF Generative Tooling Guidance this will be disclosed via the
> Generated-by trailer in the PR.
--
This message was sent by Atlassian Jira
(v10.3.26#10030026)
