I agree but you have to know in which jar a job is contained..when you
upload the jar on our application you immediately know the qualified name
of the job class and in which jar it belongs to. I think that when you
upload a jar on Flink, Flink should list all available jobs inside it
(IMHO)..it
IIUC the list of jobs contained in jar means the jobs you defined in the
pipeline. Then I don't think it is flink's responsibility to maintain the
job list info, it is the job scheduler that define the pipeline. So the job
scheduler should maintain the job list.
Flavio Pompermaier
The jobs are somehow related to each other in the sense that we have a
configurable pipeline where there are optional steps you can enable/disable
(and thus we create a single big jar).
Because of this, we have our application REST service that actually works
also as a job scheduler and use the
Thanks Flavio,
I get most of your points except one
- Get the list of jobs contained in jar (ideally this is is true for
every engine beyond Spark or Flink)
Just curious to know how you submit job via rest api, if there're multiple
jobs in one jar, then do you need to submit jar one time
Hi Jeff, the thing about the manifest is really about to have a way to list
multiple main classes in the jart (without the need to inspect every Java
class or forcing a 1-to-1 between jar and job like it is now).
My requirements were driven by the UI we're using in our framework:
- Get the
Hi Flavio,
Based on the discussion in the tickets you mentioned above, the
program-class attribute was a mistake and community is intended to use
main-class to replace it.
Deprecating Program interface is a part of work of flink new client api.
IIUC, your requirements are not so complicated. We
Hi Tison,
we use a modified version of the Program interface to enable a web UI do
properly detect and run Flink jobs contained in a jar + their parameters.
As stated in [1], we dected multiple Main classes per jar by handling an
extra comma-separeted Manifest entry (i.e. 'Main-classes').
As
Hi guys,
We want to have an accurate idea of how many people are implementing
Flink job based on the interface Program, and how they actually
implement it.
The reason I ask for the survey is from this thread[1] where we notice
this codepath is stale and less useful than it should be. As it is an
