[
https://issues.apache.org/jira/browse/FLINK-15208?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bowen Li updated FLINK-15208:
-----------------------------
Summary: client submits multiple sub-jobs for job with dynamic catalog
table (was: support client to submit multiple sub-jobs for job with dynamic
catalog table)
> client submits multiple sub-jobs for job with dynamic catalog table
> -------------------------------------------------------------------
>
> Key: FLINK-15208
> URL: https://issues.apache.org/jira/browse/FLINK-15208
> Project: Flink
> Issue Type: New Feature
> Components: Table SQL / API, Table SQL / Client
> Reporter: Bowen Li
> Assignee: Bowen Li
> Priority: Major
>
> with dynamic catalog table in FLINK-15206, users can maintain a single SQL
> job for both their online and offline job. However, they still need to change
> their configurations in order to submit different jobs over time.
> E.g. when users update logic of their streaming job, they need to bootstrap
> both a new online job and backfill offline job, let's call them sub-jobs of a
> job with dynamic catalog table. They would have to
> 1) manually change execution mode in yaml config to "streaming" and submit
> the streaming job
> 2) manually change execution mode in yaml config to "batch" and submit the
> batch job
> we should introduce a mechanism to allow users submit all or a subset of
> sub-jobs all at once. In the backfill use case mentioned above, ideally users
> should just execute the SQL once, and Flink should spin up two jobs for our
> users.
> Streaming platform at some big companies like Uber and Netflix are already
> kind of doing this for backfill use cases one way or another - some do it in
> UI, some do it in planning phase. Would be great to standardize this practice
> and provide users with ultimate simplicity.
> The assumption here is that users are fully aware of the consequences of
> launching two/multiple jobs at the same time. E.g. they need to handle
> overlapped results if there's any,
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
