Re: Flink run different jars
Thanks a lot Yangze. That is very helpful! On Tue, Sep 28, 2021 at 11:11 PM Yangze Guo wrote: > You need to edit the conf/workers. Example of the config[1] and the > process[2]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster > [2] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode > > Best, > Yangze Guo > > On Wed, Sep 29, 2021 at 1:02 PM Qihua Yang wrote: > > > > Hi Yangze, > > > > Thanks a lot for your reply. References are very helpful! > > Another quick question. Reference 1 can start a standalone cluster > (session Mode). That cluster has a jobManager. I can submit job to run. How > about taskManger? Do I need to manually start multiple taskManagers? > > Is there a complete example to show the process? > > > > Thanks, > > Qihua > > > > > > On Tue, Sep 28, 2021 at 7:02 PM Yangze Guo wrote: > >> > >> Hi, Qihua > >> > >> IIUC, what you want might be a standalone cluster[1] or session > cluster[2][3]. > >> > >> [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ > >> [2] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#session-mode > >> [3] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#session-mode > >> > >> Best, > >> Yangze Guo > >> > >> On Wed, Sep 29, 2021 at 5:57 AM Qihua Yang wrote: > >> > > >> > Hi, > >> > > >> > Is that possible to run a flink app without a job? What I am trying > to do is I build multiple jars. And switch jar to run different jobs. > >> > I am not sure if flink supports this mode. I saw rest API can upload > jar, cancel job and run a jar. > >> > Right now I can upload a jar to flink. But when I cancel a job, flink > will restart automatically. I checked log. It show below logs. Can anyone > help me out? > >> > > >> > Caused by: > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: > Application Status: CANCELED > >> > at > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:73) > >> > ... 41 common frames omitted > >> > Caused by: org.apache.flink.runtime.client.JobCancellationException: > Job was cancelled. > >> > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:149) > >> > at > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:64) > >> > ... 41 common frames omitted > >> > > >> > Thanks! >
Re: Flink run different jars
You need to edit the conf/workers. Example of the config[1] and the process[2]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-and-stopping-a-cluster [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/#starting-a-standalone-cluster-session-mode Best, Yangze Guo On Wed, Sep 29, 2021 at 1:02 PM Qihua Yang wrote: > > Hi Yangze, > > Thanks a lot for your reply. References are very helpful! > Another quick question. Reference 1 can start a standalone cluster (session > Mode). That cluster has a jobManager. I can submit job to run. How about > taskManger? Do I need to manually start multiple taskManagers? > Is there a complete example to show the process? > > Thanks, > Qihua > > > On Tue, Sep 28, 2021 at 7:02 PM Yangze Guo wrote: >> >> Hi, Qihua >> >> IIUC, what you want might be a standalone cluster[1] or session >> cluster[2][3]. >> >> [1] >> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ >> [2] >> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#session-mode >> [3] >> https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#session-mode >> >> Best, >> Yangze Guo >> >> On Wed, Sep 29, 2021 at 5:57 AM Qihua Yang wrote: >> > >> > Hi, >> > >> > Is that possible to run a flink app without a job? What I am trying to do >> > is I build multiple jars. And switch jar to run different jobs. >> > I am not sure if flink supports this mode. I saw rest API can upload jar, >> > cancel job and run a jar. >> > Right now I can upload a jar to flink. But when I cancel a job, flink will >> > restart automatically. I checked log. It show below logs. Can anyone help >> > me out? >> > >> > Caused by: >> > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: >> > Application Status: CANCELED >> > at >> > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:73) >> > ... 41 common frames omitted >> > Caused by: org.apache.flink.runtime.client.JobCancellationException: Job >> > was cancelled. >> > at >> > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:149) >> > at >> > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:64) >> > ... 41 common frames omitted >> > >> > Thanks!
Re: Flink run different jars
Hi Yangze, Thanks a lot for your reply. References are very helpful! Another quick question. Reference 1 can start a standalone cluster (session Mode). That cluster has a jobManager. I can submit job to run. How about taskManger? Do I need to manually start multiple taskManagers? Is there a complete example to show the process? Thanks, Qihua On Tue, Sep 28, 2021 at 7:02 PM Yangze Guo wrote: > Hi, Qihua > > IIUC, what you want might be a standalone cluster[1] or session > cluster[2][3]. > > [1] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ > [2] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#session-mode > [3] > https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#session-mode > > Best, > Yangze Guo > > On Wed, Sep 29, 2021 at 5:57 AM Qihua Yang wrote: > > > > Hi, > > > > Is that possible to run a flink app without a job? What I am trying to > do is I build multiple jars. And switch jar to run different jobs. > > I am not sure if flink supports this mode. I saw rest API can upload > jar, cancel job and run a jar. > > Right now I can upload a jar to flink. But when I cancel a job, flink > will restart automatically. I checked log. It show below logs. Can anyone > help me out? > > > > Caused by: > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: > Application Status: CANCELED > > at > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:73) > > ... 41 common frames omitted > > Caused by: org.apache.flink.runtime.client.JobCancellationException: Job > was cancelled. > > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:149) > > at > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:64) > > ... 41 common frames omitted > > > > Thanks! >
Re: Flink run different jars
Hi, Qihua IIUC, what you want might be a standalone cluster[1] or session cluster[2][3]. [1] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/standalone/overview/ [2] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/native_kubernetes/#session-mode [3] https://ci.apache.org/projects/flink/flink-docs-master/docs/deployment/resource-providers/yarn/#session-mode Best, Yangze Guo On Wed, Sep 29, 2021 at 5:57 AM Qihua Yang wrote: > > Hi, > > Is that possible to run a flink app without a job? What I am trying to do is > I build multiple jars. And switch jar to run different jobs. > I am not sure if flink supports this mode. I saw rest API can upload jar, > cancel job and run a jar. > Right now I can upload a jar to flink. But when I cancel a job, flink will > restart automatically. I checked log. It show below logs. Can anyone help me > out? > > Caused by: > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: > Application Status: CANCELED > at > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:73) > ... 41 common frames omitted > Caused by: org.apache.flink.runtime.client.JobCancellationException: Job was > cancelled. > at > org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:149) > at > org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:64) > ... 41 common frames omitted > > Thanks!
Flink run different jars
Hi, Is that possible to run a flink app without a job? What I am trying to do is I build multiple jars. And switch jar to run different jobs. I am not sure if flink supports this mode. I saw rest API can upload jar, cancel job and run a jar. Right now I can upload a jar to flink. But when I cancel a job, flink will restart automatically. I checked log. It show below logs. Can anyone help me out? Caused by: org.apache.flink.client.deployment.application.UnsuccessfulExecutionException: Application Status: CANCELED at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:73) ... 41 common frames omitted Caused by: org.apache.flink.runtime.client.JobCancellationException: Job was cancelled. at org.apache.flink.runtime.jobmaster.JobResult.toJobExecutionResult(JobResult.java:149) at org.apache.flink.client.deployment.application.UnsuccessfulExecutionException.fromJobResult(UnsuccessfulExecutionException.java:64) ... 41 common frames omitted Thanks!