Hi devs,
Here is a proposal to reverse the dependency from flink-streaming-java to
flink-client, for a proper
module dependency graph. Since it changes current structure, it should be
discussed publicly.
The original idea comes from that flink-streaming-java acts as an API only
module just as what
we do in its batch companion flink-java. If a Flink user want to write a
minimum DataStream
program, the only dependency should be flink-streaming java.
However, currently as it is implemented, flink-client and even
flink-runtime are transitively polluted
in when user depends on flink-streaming-java. These dependencies polluted
in as
flink-client:
- previously, ClusterClient, which is removed by FLIP-73 Executors
- accidentally, ProgramInvocationException, we just throw in place as it
is accessible.
- transitively, flink-optimizer, for one utility.
- transitively, flink-java, for several utilities.
flink-runtime:
- mainly for JobGraph generating.
With a previous discussion with @Aljoscha Krettek <aljos...@apache.org> our
goal is briefly making flink-streaming-java
an API only module. As a first step we can break the dependency from
flink-streaming-java to
flink-client[1][2].
With this first step, continuously we factor out common utilities in
flink-java to
flink-core and eventually eliminate dependencies from streaming to batch;
while
orthogonally, we factor out job compilation logic into
flink-streaming-compiler module and
break the dependency to flink-runtime. The final dependency graph will be:
flink-client -> flink-streaming-compiler -> flink-runtime
\->
flink-streaming-java
Looking forward to your feedback. Basically whether or not it is in a right
direction, and if so,
how the community integrates this proposal.
Best,
tison.
[1] https://issues.apache.org/jira/browse/FLINK-15090
[2] https://issues.apache.org/jira/browse/FLINK-16427
