pan3793 opened a new pull request, #7628:
URL: https://github.com/apache/kyuubi/pull/7628
### Why are the changes needed?
Closes #6995.
Extend the Flink SQL engine to support Flink 2.0, 2.1, 2.2 and 2.3, while
keeping the existing
1.17 to 1.20 support. The engine keeps shipping a single jar that runs
against all supported
Flink versions, so the version differences are bridged by reflection rather
than by per-version
source trees.
Flink 2.x changes that the engine has to adapt to:
- FLINK-14068 (2.0.0) removed `org.apache.flink.api.common.time.Time` in
favor of `Duration`,
which is a parameter of `EmbeddedJobClient`'s constructor.
- FLINK-33212 (2.0.0) added a `Configuration` parameter to
`EmbeddedExecutor`'s constructor.
- FLINK-38974 (2.3.0) split `EmbeddedExecutorFactory`'s and
`EmbeddedExecutor`'s job ids into the
application, suspended and terminal ones. Flink resolves the factory
constructor at compile
time, so Kyuubi's copy of `EmbeddedExecutorFactory` has to declare both
constructors.
- FLINK-35625 and FLINK-36310 (2.0.0) merged `flink run-application` into
`flink run` and removed
the former, so `FlinkProcessBuilder` picks the action by the version
detected from the
`flink-dist` jar under `$FLINK_HOME/lib`.
- FLINK-36760 (2.0.0) removed `CliOptionsParser#checkFilePath`, which is now
inlined into
`FlinkEngineUtils`. Note that `DefaultContext#load` needs no adaption, its
dependency list
changed from `List<URL>` to `List<URI>` but both erase to the same
descriptor, and Flink still
reads the elements as `URL` in `SessionContext#create`.
- `Configuration#getString(ConfigOption)` was removed,
`Configuration#get(ConfigOption)` is used
instead, which exists in all supported versions.
Flink 2.x requires Java 11 or higher and is compiled with Java 11 target, so
the `flink-2.x`
profiles raise `enforcer.maxJdkVersion` to 11, and the Flink 2.x CI jobs run
on Java 17.
Known limitation: application mode does not work on Flink 2.3, because
FLINK-38974 makes the
dispatcher reject jobs that carry no application id, while the engine
submits jobs through its
own copy of `EmbeddedExecutorFactory` which does not stamp one.
`FlinkOperationOnYarnSuite` is
skipped on Flink 2.3 accordingly, and session mode is unaffected. This is
left to a follow-up.
### How was this patch tested?
New UTs cover the Flink version detection and the application mode command
selection in
`FlinkProcessBuilderSuite`.
`FlinkOperationSuite` asserts the result of `ENCODE` by version, since
FLINK-38062 (2.2.0)
corrected its return type from `BINARY` to `VARBINARY`, before that the
result was silently
truncated to the first byte.
`externals/kyuubi-flink-sql-engine` tests were run against each Flink binary
distribution, with
`FLINK_HOME` pinned to the matching distribution:
| Flink | Result |
| --- | --- |
| 1.20.5 | 111 passed |
| 2.0.2 | 111 passed |
| 2.1.3 | 111 passed |
| 2.2.1 | 111 passed |
| 2.3.0 | 60 passed, 51 canceled (application mode) |
The GA `flink-it` matrix now always compiles against the default Flink
version and varies only
the runtime distribution, covering 1.17, 1.18, 1.19 on Java 8 and 2.0, 2.1,
2.2, 2.3 on Java 17.
### Was this patch authored or co-authored using generative AI tooling?
Assisted-by: Claude Opus 4.5
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
