rmetzger opened a new pull request #14767:
URL: https://github.com/apache/flink/pull/14767
## What is the purpose of the change
In order to distinguish between batch and streaming jobs we propose to
introduce an enum JobType which is set in the JobGraph when creating it. Using
the JobType it will be possible to decide which scheduler to use depending on
the nature of the job.
For batch jobs (from the DataSet API), setting this field is trivial (in the
JobGraphGenerator).
For streaming jobs the situation is more complicated, since FLIP-134
introduced support for bounded (batch) jobs in the DataStream API. For the
DataStream API, we rely on the result of
StreamGraphGenerator#shouldExecuteInBatchMode, which checks if the DataStream
program has unbounded sources.
Lastly, the Blink Table API / SQL Planner also generates StreamGraph
instances, which contain batch jobs. We are tagging the StreamGraph as a batch
job in the ExecutorUtils.setBatchProperties() method.
## Verifying this change
This change added tests.
## Does this pull request potentially affect one of the following parts:
- Dependencies (does it add or upgrade a dependency): (yes / **no**)
- The public API, i.e., is any changed class annotated with
`@Public(Evolving)`: (yes / **no**)
- The serializers: (yes / **no** / don't know)
- The runtime per-record code paths (performance sensitive): (yes / **no**
/ don't know)
- Anything that affects deployment or recovery: JobManager (and its
components), Checkpointing, Kubernetes/Yarn/Mesos, ZooKeeper: (yes / **no** /
don't know)
- The S3 file system connector: (yes / **no** / don't know)
## Documentation
- Does this pull request introduce a new feature? (yes / **no**)
- If yes, how is the feature documented? (not applicable / docs /
**JavaDocs** / not documented)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
