Run each job individually as described here:
https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/yarn_setup.html#run-a-single-flink-job-on-yarn
Yes they will run concurrently and be completely isolated from each other.
-Jamie
On Sun, Jan 27, 2019 at 6:08 AM Eran Twili <eran.tw...@niceactimize.com>
wrote:
> Hi,
>
>
>
> In my company we are interesting in running Flink jobs on a Yarn cluster
> (on AWS EMR).
>
> We understand that there are 2 ways ('modes') to execute Flink jobs on a
> yarn cluster.
>
> We *must have the jobs run concurrently!*
>
> From what we understand so far those are the options:
>
> 1. Start a long running yarn session, to which we'll send jobs.
> 2. Run each job as a 'single job'.
>
> We searched the web to understand the difference and consequences of each
> option,
>
> (We read threw flink-yarn-setup
> <https://ci.apache.org/projects/flink/flink-docs-release-1.7/ops/deployment/yarn_setup.html#yarn-setup>
> and FLIP6
> <https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=65147077>,
> along many other references),
>
> but couldn't find clear comprehensive info.
>
>
>
> In the 'session' mode:
>
> 1. Does running multiple jobs in single session means there's no job
> isolation?
> 2. All jobs will run on the same jvm?
> 3. Can we define different classpath for each job in this mode?
>
> In the 'single job' mode:
>
> 1. Can we run multiple jobs concurrently?
> 2. Is there a complete job isolation by default or do we need to
> configure it (different jvm/classpath)?
>
>
>
> Overall, what will be the different implications in aspects of resource
> management, security, and monitoring?
>
> Another question: what is the difference between multiple sessions of a
> single job vs multiple 'single job' executions?
>
>
>
> We'll be very thankful if someone could provide some answers or reference
> to a comprehensive documentation on those subjects.
>
>
>
> Regards,
>
> Eran
>
>
>
>
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
>
> Confidentiality: This communication and any attachments are intended for
> the above-named persons only and may be confidential and/or legally
> privileged. Any opinions expressed in this communication are not
> necessarily those of NICE Actimize. If this communication has come to you
> in error you must take no action based on it, nor must you copy or show it
> to anyone; please delete/destroy and inform the sender by e-mail
> immediately.
> Monitoring: NICE Actimize may monitor incoming and outgoing e-mails.
> Viruses: Although we have taken steps toward ensuring that this e-mail and
> attachments are free from any virus, we advise that in keeping with good
> computing practice the recipient should ensure they are actually virus free.
>
