Re: Re: [DISCUSS] Future of Per-Job Mode
Do you mean a new interface just like *ApplicationDeployer* for flink-jar-application submission? The *ApplicationDeployer*[1] interface has already been used in the flink-kubernetes-operator project. [1]. https://github.com/apache/flink-kubernetes-operator/blob/main/flink-kubernetes-operator/src/main/java/org/apache/flink/kubernetes/operator/service/FlinkService.java#L65 Best, Yang Jark Wu 于2022年2月17日周四 22:18写道： > Hi Yang, > > It would be better to have an interface to submitting application mode > jobs. It's a little tricky for SQL CLI/Gateway process to invoke a shell > script to submit jobs. > > Best, > Jark > > On Thu, 17 Feb 2022 at 15:20, Yang Wang wrote: > > > I am not sure whether the sql script could also be submitted like python. > > We will need a sql-runner jar, which plays as the user jar and has the > sql > > script as the argument. > > > > ./bin/flink run-application \ --target kubernetes-application \ > >-Dkubernetes.cluster-id= \ > > -Dkubernetes.container.image= \ --sqlFiles > > /opt/flink/examples/sql/word_count.sql > > > > Best, > > Yang > > > > Jark Wu 于2022年2月16日周三 20:00写道： > > > > > I think this mode is still limited and maybe not easy to extend. > > > Could the application mode provide an interface to execute? > > > So that clients can implement the interface and pass arbitrary > parameters > > > (e.g. SQL scripts) ? > > > > > > Best, > > > Jark > > > > > > On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf > > wrote: > > > > > > > Hi Jark, > > > > > > > > I think you are raising a very good point. I think we need an > > application > > > > mode for SQL that would work along the lines of executing a SQL > script > > > > (incl. init scripts) located in a particular directory in the Docker > > > Image. > > > > Details to be discussed. > > > > > > > > Do you think Zeppelin/SQL CLI could work with such a mode for > > > > non-interactive queries (interactive queries would use a session > > > cluster)? > > > > > > > > Best, > > > > > > > > Konstantin > > > > > > > > > > > > On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > > > > > > > > > Hi David, > > > > > > > > > > Zeppelin and SQL CLI also support submitting long-running streaming > > SQL > > > > > jobs. So the session cluster is not a fit mode. > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > On Fri, 11 Feb 2022 at 22:42, David Morávek > wrote: > > > > > > > > > > > Hi Jark, can you please elaborate about the current need of the > > > per-job > > > > > > mode for interactive clients (eg. Zeppelin that you've > mentioned)? > > > > Aren't > > > > > > these a natural fit for the session cluster? > > > > > > > > > > > > D. > > > > > > > > > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu > wrote: > > > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > > > I'm not very familiar with the implementation of per-job mode > and > > > > > > > application mode. > > > > > > > But is there any instruction for users abou how to migrate > > > > > platforms/jobs > > > > > > > to application mode? > > > > > > > IIUC, the biggest difference between the two modes is where the > > > > main() > > > > > > > method is executed. > > > > > > > However, SQL jobs are not jar applications and don't have the > > > main() > > > > > > > method. > > > > > > > For example, SQL CLI submits SQL jobs by invoking > > > > > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > > > > > How SQL Client and SQL platforms (e.g. Zeppelin) support > > > application > > > > > > mode? > > > > > > > > > > > > > > Best, > > > > > > > Jark > > > > > > > > > > > > > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf < > > kna...@apache.org> > > > > > > wrote: > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > > > > > these limitations of per-job mode on YARN. It seems that > there > > > is a > > > > > > > general > > > > > > > > agreement to deprecate per-job mode and to drop it once the > > > > > limitations > > > > > > > > around YARN are resolved. I've started a corresponding vote > in > > > [1]. > > > > > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > > > > > > > [1] > > > > https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > > > > > > > Thank you for the clarification. In general I think we will > > > have > > > > > time > > > > > > > to > > > > > > > > > experiment with this until it will be removed totally and > > > migrate > > > > > our > > > > > > > > > solution to use application mode. > > > > > > > > > > > > > > > > > > Regards, > > > > > > > > > F > > > > > > > > > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > > > > > Hi
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi Yang, It would be better to have an interface to submitting application mode jobs. It's a little tricky for SQL CLI/Gateway process to invoke a shell script to submit jobs. Best, Jark On Thu, 17 Feb 2022 at 15:20, Yang Wang wrote: > I am not sure whether the sql script could also be submitted like python. > We will need a sql-runner jar, which plays as the user jar and has the sql > script as the argument. > > ./bin/flink run-application \ --target kubernetes-application \ >-Dkubernetes.cluster-id= \ > -Dkubernetes.container.image= \ --sqlFiles > /opt/flink/examples/sql/word_count.sql > > Best, > Yang > > Jark Wu 于2022年2月16日周三 20:00写道： > > > I think this mode is still limited and maybe not easy to extend. > > Could the application mode provide an interface to execute? > > So that clients can implement the interface and pass arbitrary parameters > > (e.g. SQL scripts) ? > > > > Best, > > Jark > > > > On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf > wrote: > > > > > Hi Jark, > > > > > > I think you are raising a very good point. I think we need an > application > > > mode for SQL that would work along the lines of executing a SQL script > > > (incl. init scripts) located in a particular directory in the Docker > > Image. > > > Details to be discussed. > > > > > > Do you think Zeppelin/SQL CLI could work with such a mode for > > > non-interactive queries (interactive queries would use a session > > cluster)? > > > > > > Best, > > > > > > Konstantin > > > > > > > > > On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > > > > > > > Hi David, > > > > > > > > Zeppelin and SQL CLI also support submitting long-running streaming > SQL > > > > jobs. So the session cluster is not a fit mode. > > > > > > > > Best, > > > > Jark > > > > > > > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > > > > > > > Hi Jark, can you please elaborate about the current need of the > > per-job > > > > > mode for interactive clients (eg. Zeppelin that you've mentioned)? > > > Aren't > > > > > these a natural fit for the session cluster? > > > > > > > > > > D. > > > > > > > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > I'm not very familiar with the implementation of per-job mode and > > > > > > application mode. > > > > > > But is there any instruction for users abou how to migrate > > > > platforms/jobs > > > > > > to application mode? > > > > > > IIUC, the biggest difference between the two modes is where the > > > main() > > > > > > method is executed. > > > > > > However, SQL jobs are not jar applications and don't have the > > main() > > > > > > method. > > > > > > For example, SQL CLI submits SQL jobs by invoking > > > > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > > > > How SQL Client and SQL platforms (e.g. Zeppelin) support > > application > > > > > mode? > > > > > > > > > > > > Best, > > > > > > Jark > > > > > > > > > > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf < > kna...@apache.org> > > > > > wrote: > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > > > > these limitations of per-job mode on YARN. It seems that there > > is a > > > > > > general > > > > > > > agreement to deprecate per-job mode and to drop it once the > > > > limitations > > > > > > > around YARN are resolved. I've started a corresponding vote in > > [1]. > > > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > > > > [1] > > > https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > > > > > Thank you for the clarification. In general I think we will > > have > > > > time > > > > > > to > > > > > > > > experiment with this until it will be removed totally and > > migrate > > > > our > > > > > > > > solution to use application mode. > > > > > > > > > > > > > > > > Regards, > > > > > > > > F > > > > > > > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > > > > Hi all, > > > > > > > > > > > > > > > > > > I remember the application mode was initially named > "cluster > > > > mode". > > > > > > As > > > > > > > a > > > > > > > > > contrast, the per-job mode is the "client mode". > > > > > > > > > So I believe application mode should cover all the > > > > functionalities > > > > > of > > > > > > > > > per-job except where we are running the user main code. > > > > > > > > > In the containerized or the Kubernetes world, the > application > > > > mode > > > > > is > > > > > > > > more > > > > > > > > > native and easy to use since all the Flink and user > > > > > > > > > jars are bundled in the image. I am also in favor of > > > deprecating > > > > > and > > > > > > > > > removing
Re: Re: [DISCUSS] Future of Per-Job Mode
I am not sure whether the sql script could also be submitted like python. We will need a sql-runner jar, which plays as the user jar and has the sql script as the argument. ./bin/flink run-application \ --target kubernetes-application \ -Dkubernetes.cluster-id= \ -Dkubernetes.container.image= \ --sqlFiles /opt/flink/examples/sql/word_count.sql Best, Yang Jark Wu 于2022年2月16日周三 20:00写道： > I think this mode is still limited and maybe not easy to extend. > Could the application mode provide an interface to execute? > So that clients can implement the interface and pass arbitrary parameters > (e.g. SQL scripts) ? > > Best, > Jark > > On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf wrote: > > > Hi Jark, > > > > I think you are raising a very good point. I think we need an application > > mode for SQL that would work along the lines of executing a SQL script > > (incl. init scripts) located in a particular directory in the Docker > Image. > > Details to be discussed. > > > > Do you think Zeppelin/SQL CLI could work with such a mode for > > non-interactive queries (interactive queries would use a session > cluster)? > > > > Best, > > > > Konstantin > > > > > > On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > > > > > Hi David, > > > > > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > > > jobs. So the session cluster is not a fit mode. > > > > > > Best, > > > Jark > > > > > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > > > > > Hi Jark, can you please elaborate about the current need of the > per-job > > > > mode for interactive clients (eg. Zeppelin that you've mentioned)? > > Aren't > > > > these a natural fit for the session cluster? > > > > > > > > D. > > > > > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > > > > > Hi Konstantin, > > > > > > > > > > I'm not very familiar with the implementation of per-job mode and > > > > > application mode. > > > > > But is there any instruction for users abou how to migrate > > > platforms/jobs > > > > > to application mode? > > > > > IIUC, the biggest difference between the two modes is where the > > main() > > > > > method is executed. > > > > > However, SQL jobs are not jar applications and don't have the > main() > > > > > method. > > > > > For example, SQL CLI submits SQL jobs by invoking > > > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > > > How SQL Client and SQL platforms (e.g. Zeppelin) support > application > > > > mode? > > > > > > > > > > Best, > > > > > Jark > > > > > > > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > > > wrote: > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > > > these limitations of per-job mode on YARN. It seems that there > is a > > > > > general > > > > > > agreement to deprecate per-job mode and to drop it once the > > > limitations > > > > > > around YARN are resolved. I've started a corresponding vote in > [1]. > > > > > > > > > > > > Thanks again, > > > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > > > > [1] > > https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > > > > > > > wrote: > > > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > > > Thank you for the clarification. In general I think we will > have > > > time > > > > > to > > > > > > > experiment with this until it will be removed totally and > migrate > > > our > > > > > > > solution to use application mode. > > > > > > > > > > > > > > Regards, > > > > > > > F > > > > > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > > > Hi all, > > > > > > > > > > > > > > > > I remember the application mode was initially named "cluster > > > mode". > > > > > As > > > > > > a > > > > > > > > contrast, the per-job mode is the "client mode". > > > > > > > > So I believe application mode should cover all the > > > functionalities > > > > of > > > > > > > > per-job except where we are running the user main code. > > > > > > > > In the containerized or the Kubernetes world, the application > > > mode > > > > is > > > > > > > more > > > > > > > > native and easy to use since all the Flink and user > > > > > > > > jars are bundled in the image. I am also in favor of > > deprecating > > > > and > > > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > > > IIRC, the YARN application mode could ship user jars and > > > > dependencies > > > > > > via > > > > > > > > "yarn.ship-files" config option. The only > > > > > > > > limitation is that we could not ship and load the user > > > dependencies > > > > > > with > > > > > > > > user classloader, not the parent classloader. > > > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > > > directory > > > > > > > > automatically. > > >
Re: Re: [DISCUSS] Future of Per-Job Mode
I think this mode is still limited and maybe not easy to extend. Could the application mode provide an interface to execute? So that clients can implement the interface and pass arbitrary parameters (e.g. SQL scripts) ? Best, Jark On Wed, 16 Feb 2022 at 18:54, Konstantin Knauf wrote: > Hi Jark, > > I think you are raising a very good point. I think we need an application > mode for SQL that would work along the lines of executing a SQL script > (incl. init scripts) located in a particular directory in the Docker Image. > Details to be discussed. > > Do you think Zeppelin/SQL CLI could work with such a mode for > non-interactive queries (interactive queries would use a session cluster)? > > Best, > > Konstantin > > > On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > > > Hi David, > > > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > > jobs. So the session cluster is not a fit mode. > > > > Best, > > Jark > > > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > > > Hi Jark, can you please elaborate about the current need of the per-job > > > mode for interactive clients (eg. Zeppelin that you've mentioned)? > Aren't > > > these a natural fit for the session cluster? > > > > > > D. > > > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > > > Hi Konstantin, > > > > > > > > I'm not very familiar with the implementation of per-job mode and > > > > application mode. > > > > But is there any instruction for users abou how to migrate > > platforms/jobs > > > > to application mode? > > > > IIUC, the biggest difference between the two modes is where the > main() > > > > method is executed. > > > > However, SQL jobs are not jar applications and don't have the main() > > > > method. > > > > For example, SQL CLI submits SQL jobs by invoking > > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > > How SQL Client and SQL platforms (e.g. Zeppelin) support application > > > mode? > > > > > > > > Best, > > > > Jark > > > > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > > wrote: > > > > > > > > > Hi everyone, > > > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > > these limitations of per-job mode on YARN. It seems that there is a > > > > general > > > > > agreement to deprecate per-job mode and to drop it once the > > limitations > > > > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > > > > > > > Thanks again, > > > > > > > > > > Konstantin > > > > > > > > > > > > > > > [1] > https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > > > > wrote: > > > > > > > > > > > Hi Yang, > > > > > > > > > > > > Thank you for the clarification. In general I think we will have > > time > > > > to > > > > > > experiment with this until it will be removed totally and migrate > > our > > > > > > solution to use application mode. > > > > > > > > > > > > Regards, > > > > > > F > > > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > > Hi all, > > > > > > > > > > > > > > I remember the application mode was initially named "cluster > > mode". > > > > As > > > > > a > > > > > > > contrast, the per-job mode is the "client mode". > > > > > > > So I believe application mode should cover all the > > functionalities > > > of > > > > > > > per-job except where we are running the user main code. > > > > > > > In the containerized or the Kubernetes world, the application > > mode > > > is > > > > > > more > > > > > > > native and easy to use since all the Flink and user > > > > > > > jars are bundled in the image. I am also in favor of > deprecating > > > and > > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > > IIRC, the YARN application mode could ship user jars and > > > dependencies > > > > > via > > > > > > > "yarn.ship-files" config option. The only > > > > > > > limitation is that we could not ship and load the user > > dependencies > > > > > with > > > > > > > user classloader, not the parent classloader. > > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > > directory > > > > > > > automatically. > > > > > > > > > > > > > > > > > > > > > Best, > > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > > > > > First of all, sorry for the delay. We at Cloudera are > currently > > > > > > relying on > > > > > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > > > > > > > > > Specifically, we allow users to upload connector jars and > other > > > > > > artifacts. > > > > > > > > There are also some default jars that we need to ship. These > > are > > > > all > > > > > > stored > > > > > > > > on the local file system of our service’s node. The Flink
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi Jark, I think you are raising a very good point. I think we need an application mode for SQL that would work along the lines of executing a SQL script (incl. init scripts) located in a particular directory in the Docker Image. Details to be discussed. Do you think Zeppelin/SQL CLI could work with such a mode for non-interactive queries (interactive queries would use a session cluster)? Best, Konstantin On Sat, Feb 12, 2022 at 4:31 AM Jark Wu wrote: > Hi David, > > Zeppelin and SQL CLI also support submitting long-running streaming SQL > jobs. So the session cluster is not a fit mode. > > Best, > Jark > > On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > > > Hi Jark, can you please elaborate about the current need of the per-job > > mode for interactive clients (eg. Zeppelin that you've mentioned)? Aren't > > these a natural fit for the session cluster? > > > > D. > > > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > > > Hi Konstantin, > > > > > > I'm not very familiar with the implementation of per-job mode and > > > application mode. > > > But is there any instruction for users abou how to migrate > platforms/jobs > > > to application mode? > > > IIUC, the biggest difference between the two modes is where the main() > > > method is executed. > > > However, SQL jobs are not jar applications and don't have the main() > > > method. > > > For example, SQL CLI submits SQL jobs by invoking > > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > > How SQL Client and SQL platforms (e.g. Zeppelin) support application > > mode? > > > > > > Best, > > > Jark > > > > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > > wrote: > > > > > > > Hi everyone, > > > > > > > > Thank you for sharing your perspectives. I was not aware of > > > > these limitations of per-job mode on YARN. It seems that there is a > > > general > > > > agreement to deprecate per-job mode and to drop it once the > limitations > > > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > > > > > Thanks again, > > > > > > > > Konstantin > > > > > > > > > > > > [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > > > > wrote: > > > > > > > > > Hi Yang, > > > > > > > > > > Thank you for the clarification. In general I think we will have > time > > > to > > > > > experiment with this until it will be removed totally and migrate > our > > > > > solution to use application mode. > > > > > > > > > > Regards, > > > > > F > > > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > > Hi all, > > > > > > > > > > > > I remember the application mode was initially named "cluster > mode". > > > As > > > > a > > > > > > contrast, the per-job mode is the "client mode". > > > > > > So I believe application mode should cover all the > functionalities > > of > > > > > > per-job except where we are running the user main code. > > > > > > In the containerized or the Kubernetes world, the application > mode > > is > > > > > more > > > > > > native and easy to use since all the Flink and user > > > > > > jars are bundled in the image. I am also in favor of deprecating > > and > > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > > IIRC, the YARN application mode could ship user jars and > > dependencies > > > > via > > > > > > "yarn.ship-files" config option. The only > > > > > > limitation is that we could not ship and load the user > dependencies > > > > with > > > > > > user classloader, not the parent classloader. > > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" > directory > > > > > > automatically. > > > > > > > > > > > > > > > > > > Best, > > > > > > Yang > > > > > > > > > > > > > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > > > First of all, sorry for the delay. We at Cloudera are currently > > > > > relying on > > > > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > > > > > > > Specifically, we allow users to upload connector jars and other > > > > > artifacts. > > > > > > > There are also some default jars that we need to ship. These > are > > > all > > > > > stored > > > > > > > on the local file system of our service’s node. The Flink job > is > > > > > submitted > > > > > > > on the users’ behalf by our service, which also specifies the > > jars > > > to > > > > > ship. > > > > > > > The service runs on a single node, not on all nodes with Flink > > > TM/JM. > > > > > It > > > > > > > would thus be difficult to manage the jars on every node. > > > > > > > > > > > > > > We are not familiar with the reasoning behind why application > > mode > > > > > > > currently doesn’t ship the user jars, besides the deployment > > being > > > > > faster > > > > > > > this way. Would it be possible for the application mode to > >
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi David, Zeppelin and SQL CLI also support submitting long-running streaming SQL jobs. So the session cluster is not a fit mode. Best, Jark On Fri, 11 Feb 2022 at 22:42, David Morávek wrote: > Hi Jark, can you please elaborate about the current need of the per-job > mode for interactive clients (eg. Zeppelin that you've mentioned)? Aren't > these a natural fit for the session cluster? > > D. > > On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > > > Hi Konstantin, > > > > I'm not very familiar with the implementation of per-job mode and > > application mode. > > But is there any instruction for users abou how to migrate platforms/jobs > > to application mode? > > IIUC, the biggest difference between the two modes is where the main() > > method is executed. > > However, SQL jobs are not jar applications and don't have the main() > > method. > > For example, SQL CLI submits SQL jobs by invoking > > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > > How SQL Client and SQL platforms (e.g. Zeppelin) support application > mode? > > > > Best, > > Jark > > > > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf > wrote: > > > > > Hi everyone, > > > > > > Thank you for sharing your perspectives. I was not aware of > > > these limitations of per-job mode on YARN. It seems that there is a > > general > > > agreement to deprecate per-job mode and to drop it once the limitations > > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > > > Thanks again, > > > > > > Konstantin > > > > > > > > > [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > > > > wrote: > > > > > > > Hi Yang, > > > > > > > > Thank you for the clarification. In general I think we will have time > > to > > > > experiment with this until it will be removed totally and migrate our > > > > solution to use application mode. > > > > > > > > Regards, > > > > F > > > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > > Hi all, > > > > > > > > > > I remember the application mode was initially named "cluster mode". > > As > > > a > > > > > contrast, the per-job mode is the "client mode". > > > > > So I believe application mode should cover all the functionalities > of > > > > > per-job except where we are running the user main code. > > > > > In the containerized or the Kubernetes world, the application mode > is > > > > more > > > > > native and easy to use since all the Flink and user > > > > > jars are bundled in the image. I am also in favor of deprecating > and > > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > > > > > @Ferenc > > > > > IIRC, the YARN application mode could ship user jars and > dependencies > > > via > > > > > "yarn.ship-files" config option. The only > > > > > limitation is that we could not ship and load the user dependencies > > > with > > > > > user classloader, not the parent classloader. > > > > > FLINK-24897 is trying to fix this via supporting "usrlib" directory > > > > > automatically. > > > > > > > > > > > > > > > Best, > > > > > Yang > > > > > > > > > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > > > > > Hi Konstantin, > > > > > > > > > > > > First of all, sorry for the delay. We at Cloudera are currently > > > > relying on > > > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > > > > > Specifically, we allow users to upload connector jars and other > > > > artifacts. > > > > > > There are also some default jars that we need to ship. These are > > all > > > > stored > > > > > > on the local file system of our service’s node. The Flink job is > > > > submitted > > > > > > on the users’ behalf by our service, which also specifies the > jars > > to > > > > ship. > > > > > > The service runs on a single node, not on all nodes with Flink > > TM/JM. > > > > It > > > > > > would thus be difficult to manage the jars on every node. > > > > > > > > > > > > We are not familiar with the reasoning behind why application > mode > > > > > > currently doesn’t ship the user jars, besides the deployment > being > > > > faster > > > > > > this way. Would it be possible for the application mode to > > > (optionally, > > > > > > enabled by some config) distribute these, or are there some > > technical > > > > > > limitations? > > > > > > > > > > > > For us it would be crucial to achieve the functionality we have > at > > > the > > > > > > moment over YARN. We started to track > > > > > > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > > > > > > mentioned as well. > > > > > > > > > > > > Considering the above, for us the more soonish removal does not > > sound > > > > > > really well. We can live with this feature as deprecated of > course, > > > > but it > > > > > > would be nice to have some time to figure out how we can utilize > > > > > > Application Mode exactly and make necessary changes if required. > > > > > > > > > >
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi Jark, can you please elaborate about the current need of the per-job mode for interactive clients (eg. Zeppelin that you've mentioned)? Aren't these a natural fit for the session cluster? D. On Fri, Feb 11, 2022 at 3:25 PM Jark Wu wrote: > Hi Konstantin, > > I'm not very familiar with the implementation of per-job mode and > application mode. > But is there any instruction for users abou how to migrate platforms/jobs > to application mode? > IIUC, the biggest difference between the two modes is where the main() > method is executed. > However, SQL jobs are not jar applications and don't have the main() > method. > For example, SQL CLI submits SQL jobs by invoking > `StreamExecutionEnvironment#executeAsync(StreamGraph)`. > How SQL Client and SQL platforms (e.g. Zeppelin) support application mode? > > Best, > Jark > > > On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf wrote: > > > Hi everyone, > > > > Thank you for sharing your perspectives. I was not aware of > > these limitations of per-job mode on YARN. It seems that there is a > general > > agreement to deprecate per-job mode and to drop it once the limitations > > around YARN are resolved. I've started a corresponding vote in [1]. > > > > Thanks again, > > > > Konstantin > > > > > > [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > > > wrote: > > > > > Hi Yang, > > > > > > Thank you for the clarification. In general I think we will have time > to > > > experiment with this until it will be removed totally and migrate our > > > solution to use application mode. > > > > > > Regards, > > > F > > > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > > Hi all, > > > > > > > > I remember the application mode was initially named "cluster mode". > As > > a > > > > contrast, the per-job mode is the "client mode". > > > > So I believe application mode should cover all the functionalities of > > > > per-job except where we are running the user main code. > > > > In the containerized or the Kubernetes world, the application mode is > > > more > > > > native and easy to use since all the Flink and user > > > > jars are bundled in the image. I am also in favor of deprecating and > > > > removing the per-job in the long run. > > > > > > > > > > > > > > > > @Ferenc > > > > IIRC, the YARN application mode could ship user jars and dependencies > > via > > > > "yarn.ship-files" config option. The only > > > > limitation is that we could not ship and load the user dependencies > > with > > > > user classloader, not the parent classloader. > > > > FLINK-24897 is trying to fix this via supporting "usrlib" directory > > > > automatically. > > > > > > > > > > > > Best, > > > > Yang > > > > > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > > > Hi Konstantin, > > > > > > > > > > First of all, sorry for the delay. We at Cloudera are currently > > > relying on > > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > > > Specifically, we allow users to upload connector jars and other > > > artifacts. > > > > > There are also some default jars that we need to ship. These are > all > > > stored > > > > > on the local file system of our service’s node. The Flink job is > > > submitted > > > > > on the users’ behalf by our service, which also specifies the jars > to > > > ship. > > > > > The service runs on a single node, not on all nodes with Flink > TM/JM. > > > It > > > > > would thus be difficult to manage the jars on every node. > > > > > > > > > > We are not familiar with the reasoning behind why application mode > > > > > currently doesn’t ship the user jars, besides the deployment being > > > faster > > > > > this way. Would it be possible for the application mode to > > (optionally, > > > > > enabled by some config) distribute these, or are there some > technical > > > > > limitations? > > > > > > > > > > For us it would be crucial to achieve the functionality we have at > > the > > > > > moment over YARN. We started to track > > > > > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > > > > > mentioned as well. > > > > > > > > > > Considering the above, for us the more soonish removal does not > sound > > > > > really well. We can live with this feature as deprecated of course, > > > but it > > > > > would be nice to have some time to figure out how we can utilize > > > > > Application Mode exactly and make necessary changes if required. > > > > > > > > > > Thank you, > > > > > F > > > > > > > > > > On 2022/01/13 08:30:48 Konstantin Knauf wrote: > > > > > > Hi everyone, > > > > > > > > > > > > I would like to discuss and understand if the benefits of having > > > Per-Job > > > > > > Mode in Apache Flink outweigh its drawbacks. > > > > > > > > > > > > > > > > > > *# Background: Flink's Deployment Modes* > > > > > > Flink currently has three deployment modes. They differ in the > > > following > > > > > > dimensions: > > > > > > *
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi Konstantin, I'm not very familiar with the implementation of per-job mode and application mode. But is there any instruction for users abou how to migrate platforms/jobs to application mode? IIUC, the biggest difference between the two modes is where the main() method is executed. However, SQL jobs are not jar applications and don't have the main() method. For example, SQL CLI submits SQL jobs by invoking `StreamExecutionEnvironment#executeAsync(StreamGraph)`. How SQL Client and SQL platforms (e.g. Zeppelin) support application mode? Best, Jark On Fri, 28 Jan 2022 at 23:33, Konstantin Knauf wrote: > Hi everyone, > > Thank you for sharing your perspectives. I was not aware of > these limitations of per-job mode on YARN. It seems that there is a general > agreement to deprecate per-job mode and to drop it once the limitations > around YARN are resolved. I've started a corresponding vote in [1]. > > Thanks again, > > Konstantin > > > [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 > > On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky > wrote: > > > Hi Yang, > > > > Thank you for the clarification. In general I think we will have time to > > experiment with this until it will be removed totally and migrate our > > solution to use application mode. > > > > Regards, > > F > > > > On 2022/01/26 02:42:24 Yang Wang wrote: > > > Hi all, > > > > > > I remember the application mode was initially named "cluster mode". As > a > > > contrast, the per-job mode is the "client mode". > > > So I believe application mode should cover all the functionalities of > > > per-job except where we are running the user main code. > > > In the containerized or the Kubernetes world, the application mode is > > more > > > native and easy to use since all the Flink and user > > > jars are bundled in the image. I am also in favor of deprecating and > > > removing the per-job in the long run. > > > > > > > > > > > > @Ferenc > > > IIRC, the YARN application mode could ship user jars and dependencies > via > > > "yarn.ship-files" config option. The only > > > limitation is that we could not ship and load the user dependencies > with > > > user classloader, not the parent classloader. > > > FLINK-24897 is trying to fix this via supporting "usrlib" directory > > > automatically. > > > > > > > > > Best, > > > Yang > > > > > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > > > Hi Konstantin, > > > > > > > > First of all, sorry for the delay. We at Cloudera are currently > > relying on > > > > per-job mode deploying Flink applications over YARN. > > > > > > > > Specifically, we allow users to upload connector jars and other > > artifacts. > > > > There are also some default jars that we need to ship. These are all > > stored > > > > on the local file system of our service’s node. The Flink job is > > submitted > > > > on the users’ behalf by our service, which also specifies the jars to > > ship. > > > > The service runs on a single node, not on all nodes with Flink TM/JM. > > It > > > > would thus be difficult to manage the jars on every node. > > > > > > > > We are not familiar with the reasoning behind why application mode > > > > currently doesn’t ship the user jars, besides the deployment being > > faster > > > > this way. Would it be possible for the application mode to > (optionally, > > > > enabled by some config) distribute these, or are there some technical > > > > limitations? > > > > > > > > For us it would be crucial to achieve the functionality we have at > the > > > > moment over YARN. We started to track > > > > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > > > > mentioned as well. > > > > > > > > Considering the above, for us the more soonish removal does not sound > > > > really well. We can live with this feature as deprecated of course, > > but it > > > > would be nice to have some time to figure out how we can utilize > > > > Application Mode exactly and make necessary changes if required. > > > > > > > > Thank you, > > > > F > > > > > > > > On 2022/01/13 08:30:48 Konstantin Knauf wrote: > > > > > Hi everyone, > > > > > > > > > > I would like to discuss and understand if the benefits of having > > Per-Job > > > > > Mode in Apache Flink outweigh its drawbacks. > > > > > > > > > > > > > > > *# Background: Flink's Deployment Modes* > > > > > Flink currently has three deployment modes. They differ in the > > following > > > > > dimensions: > > > > > * main() method executed on Jobmanager or Client > > > > > * dependencies shipped by client or bundled with all nodes > > > > > * number of jobs per cluster & relationship between job and cluster > > > > > lifecycle* (supported resource providers) > > > > > > > > > > ## Application Mode > > > > > * main() method executed on Jobmanager > > > > > * dependencies already need to be available on all nodes > > > > > * dedicated cluster for all jobs executed from the same > main()-method > > > > > (Note: applications with more than
Re: Re: [DISCUSS] Future of Per-Job Mode
Hi everyone, Thank you for sharing your perspectives. I was not aware of these limitations of per-job mode on YARN. It seems that there is a general agreement to deprecate per-job mode and to drop it once the limitations around YARN are resolved. I've started a corresponding vote in [1]. Thanks again, Konstantin [1] https://lists.apache.org/thread/v6oz92dfp95qcox45l0f8393089oyjv4 On Fri, Jan 28, 2022 at 1:53 PM Ferenc Csaky wrote: > Hi Yang, > > Thank you for the clarification. In general I think we will have time to > experiment with this until it will be removed totally and migrate our > solution to use application mode. > > Regards, > F > > On 2022/01/26 02:42:24 Yang Wang wrote: > > Hi all, > > > > I remember the application mode was initially named "cluster mode". As a > > contrast, the per-job mode is the "client mode". > > So I believe application mode should cover all the functionalities of > > per-job except where we are running the user main code. > > In the containerized or the Kubernetes world, the application mode is > more > > native and easy to use since all the Flink and user > > jars are bundled in the image. I am also in favor of deprecating and > > removing the per-job in the long run. > > > > > > > > @Ferenc > > IIRC, the YARN application mode could ship user jars and dependencies via > > "yarn.ship-files" config option. The only > > limitation is that we could not ship and load the user dependencies with > > user classloader, not the parent classloader. > > FLINK-24897 is trying to fix this via supporting "usrlib" directory > > automatically. > > > > > > Best, > > Yang > > > > > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > > > Hi Konstantin, > > > > > > First of all, sorry for the delay. We at Cloudera are currently > relying on > > > per-job mode deploying Flink applications over YARN. > > > > > > Specifically, we allow users to upload connector jars and other > artifacts. > > > There are also some default jars that we need to ship. These are all > stored > > > on the local file system of our service’s node. The Flink job is > submitted > > > on the users’ behalf by our service, which also specifies the jars to > ship. > > > The service runs on a single node, not on all nodes with Flink TM/JM. > It > > > would thus be difficult to manage the jars on every node. > > > > > > We are not familiar with the reasoning behind why application mode > > > currently doesn’t ship the user jars, besides the deployment being > faster > > > this way. Would it be possible for the application mode to (optionally, > > > enabled by some config) distribute these, or are there some technical > > > limitations? > > > > > > For us it would be crucial to achieve the functionality we have at the > > > moment over YARN. We started to track > > > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > > > mentioned as well. > > > > > > Considering the above, for us the more soonish removal does not sound > > > really well. We can live with this feature as deprecated of course, > but it > > > would be nice to have some time to figure out how we can utilize > > > Application Mode exactly and make necessary changes if required. > > > > > > Thank you, > > > F > > > > > > On 2022/01/13 08:30:48 Konstantin Knauf wrote: > > > > Hi everyone, > > > > > > > > I would like to discuss and understand if the benefits of having > Per-Job > > > > Mode in Apache Flink outweigh its drawbacks. > > > > > > > > > > > > *# Background: Flink's Deployment Modes* > > > > Flink currently has three deployment modes. They differ in the > following > > > > dimensions: > > > > * main() method executed on Jobmanager or Client > > > > * dependencies shipped by client or bundled with all nodes > > > > * number of jobs per cluster & relationship between job and cluster > > > > lifecycle* (supported resource providers) > > > > > > > > ## Application Mode > > > > * main() method executed on Jobmanager > > > > * dependencies already need to be available on all nodes > > > > * dedicated cluster for all jobs executed from the same main()-method > > > > (Note: applications with more than one job, currently still > significant > > > > limitations like missing high-availability). Technically, a session > > > cluster > > > > dedicated to all jobs submitted from the same main() method. > > > > * supported by standalone, native kubernetes, YARN > > > > > > > > ## Session Mode > > > > * main() method executed in client > > > > * dependencies are distributed from and by the client to all nodes > > > > * cluster is shared by multiple jobs submitted from different > clients, > > > > independent lifecycle > > > > * supported by standalone, Native Kubernetes, YARN > > > > > > > > ## Per-Job Mode > > > > * main() method executed in client > > > > * dependencies are distributed from and by the client to all nodes > > > > * dedicated cluster for a single job > > > > * supported by YARN only > > > > > > > > > > > > *# Reasons to Keep** There
RE: Re: [DISCUSS] Future of Per-Job Mode
Hi Yang, Thank you for the clarification. In general I think we will have time to experiment with this until it will be removed totally and migrate our solution to use application mode. Regards, F On 2022/01/26 02:42:24 Yang Wang wrote: > Hi all, > > I remember the application mode was initially named "cluster mode". As a > contrast, the per-job mode is the "client mode". > So I believe application mode should cover all the functionalities of > per-job except where we are running the user main code. > In the containerized or the Kubernetes world, the application mode is more > native and easy to use since all the Flink and user > jars are bundled in the image. I am also in favor of deprecating and > removing the per-job in the long run. > > > > @Ferenc > IIRC, the YARN application mode could ship user jars and dependencies via > "yarn.ship-files" config option. The only > limitation is that we could not ship and load the user dependencies with > user classloader, not the parent classloader. > FLINK-24897 is trying to fix this via supporting "usrlib" directory > automatically. > > > Best, > Yang > > > > Ferenc Csaky 于2022年1月25日周二 22:05写道： > > > Hi Konstantin, > > > > First of all, sorry for the delay. We at Cloudera are currently relying on > > per-job mode deploying Flink applications over YARN. > > > > Specifically, we allow users to upload connector jars and other artifacts. > > There are also some default jars that we need to ship. These are all stored > > on the local file system of our service’s node. The Flink job is submitted > > on the users’ behalf by our service, which also specifies the jars to ship. > > The service runs on a single node, not on all nodes with Flink TM/JM. It > > would thus be difficult to manage the jars on every node. > > > > We are not familiar with the reasoning behind why application mode > > currently doesn’t ship the user jars, besides the deployment being faster > > this way. Would it be possible for the application mode to (optionally, > > enabled by some config) distribute these, or are there some technical > > limitations? > > > > For us it would be crucial to achieve the functionality we have at the > > moment over YARN. We started to track > > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > > mentioned as well. > > > > Considering the above, for us the more soonish removal does not sound > > really well. We can live with this feature as deprecated of course, but it > > would be nice to have some time to figure out how we can utilize > > Application Mode exactly and make necessary changes if required. > > > > Thank you, > > F > > > > On 2022/01/13 08:30:48 Konstantin Knauf wrote: > > > Hi everyone, > > > > > > I would like to discuss and understand if the benefits of having Per-Job > > > Mode in Apache Flink outweigh its drawbacks. > > > > > > > > > *# Background: Flink's Deployment Modes* > > > Flink currently has three deployment modes. They differ in the following > > > dimensions: > > > * main() method executed on Jobmanager or Client > > > * dependencies shipped by client or bundled with all nodes > > > * number of jobs per cluster & relationship between job and cluster > > > lifecycle* (supported resource providers) > > > > > > ## Application Mode > > > * main() method executed on Jobmanager > > > * dependencies already need to be available on all nodes > > > * dedicated cluster for all jobs executed from the same main()-method > > > (Note: applications with more than one job, currently still significant > > > limitations like missing high-availability). Technically, a session > > cluster > > > dedicated to all jobs submitted from the same main() method. > > > * supported by standalone, native kubernetes, YARN > > > > > > ## Session Mode > > > * main() method executed in client > > > * dependencies are distributed from and by the client to all nodes > > > * cluster is shared by multiple jobs submitted from different clients, > > > independent lifecycle > > > * supported by standalone, Native Kubernetes, YARN > > > > > > ## Per-Job Mode > > > * main() method executed in client > > > * dependencies are distributed from and by the client to all nodes > > > * dedicated cluster for a single job > > > * supported by YARN only > > > > > > > > > *# Reasons to Keep** There are use cases where you might need the > > > combination of a single job per cluster, but main() method execution in > > the > > > client. This combination is only supported by per-job mode. > > > * It currently exists. Existing users will need to migrate to either > > > session or application mode. > > > > > > > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > > > modes that for most users probably do the same thing. Specifically, for > > > those users that don't care where the main() method is executed and want > > to > > > submit a single job per cluster. Having two ways to do the same thing is > > > confusing. > > > * Per-Job Mode is only
Re: [DISCUSS] Future of Per-Job Mode
Hi all, I remember the application mode was initially named "cluster mode". As a contrast, the per-job mode is the "client mode". So I believe application mode should cover all the functionalities of per-job except where we are running the user main code. In the containerized or the Kubernetes world, the application mode is more native and easy to use since all the Flink and user jars are bundled in the image. I am also in favor of deprecating and removing the per-job in the long run. @Ferenc IIRC, the YARN application mode could ship user jars and dependencies via "yarn.ship-files" config option. The only limitation is that we could not ship and load the user dependencies with user classloader, not the parent classloader. FLINK-24897 is trying to fix this via supporting "usrlib" directory automatically. Best, Yang Ferenc Csaky 于2022年1月25日周二 22:05写道： > Hi Konstantin, > > First of all, sorry for the delay. We at Cloudera are currently relying on > per-job mode deploying Flink applications over YARN. > > Specifically, we allow users to upload connector jars and other artifacts. > There are also some default jars that we need to ship. These are all stored > on the local file system of our service’s node. The Flink job is submitted > on the users’ behalf by our service, which also specifies the jars to ship. > The service runs on a single node, not on all nodes with Flink TM/JM. It > would thus be difficult to manage the jars on every node. > > We are not familiar with the reasoning behind why application mode > currently doesn’t ship the user jars, besides the deployment being faster > this way. Would it be possible for the application mode to (optionally, > enabled by some config) distribute these, or are there some technical > limitations? > > For us it would be crucial to achieve the functionality we have at the > moment over YARN. We started to track > https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng > mentioned as well. > > Considering the above, for us the more soonish removal does not sound > really well. We can live with this feature as deprecated of course, but it > would be nice to have some time to figure out how we can utilize > Application Mode exactly and make necessary changes if required. > > Thank you, > F > > On 2022/01/13 08:30:48 Konstantin Knauf wrote: > > Hi everyone, > > > > I would like to discuss and understand if the benefits of having Per-Job > > Mode in Apache Flink outweigh its drawbacks. > > > > > > *# Background: Flink's Deployment Modes* > > Flink currently has three deployment modes. They differ in the following > > dimensions: > > * main() method executed on Jobmanager or Client > > * dependencies shipped by client or bundled with all nodes > > * number of jobs per cluster & relationship between job and cluster > > lifecycle* (supported resource providers) > > > > ## Application Mode > > * main() method executed on Jobmanager > > * dependencies already need to be available on all nodes > > * dedicated cluster for all jobs executed from the same main()-method > > (Note: applications with more than one job, currently still significant > > limitations like missing high-availability). Technically, a session > cluster > > dedicated to all jobs submitted from the same main() method. > > * supported by standalone, native kubernetes, YARN > > > > ## Session Mode > > * main() method executed in client > > * dependencies are distributed from and by the client to all nodes > > * cluster is shared by multiple jobs submitted from different clients, > > independent lifecycle > > * supported by standalone, Native Kubernetes, YARN > > > > ## Per-Job Mode > > * main() method executed in client > > * dependencies are distributed from and by the client to all nodes > > * dedicated cluster for a single job > > * supported by YARN only > > > > > > *# Reasons to Keep** There are use cases where you might need the > > combination of a single job per cluster, but main() method execution in > the > > client. This combination is only supported by per-job mode. > > * It currently exists. Existing users will need to migrate to either > > session or application mode. > > > > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > > modes that for most users probably do the same thing. Specifically, for > > those users that don't care where the main() method is executed and want > to > > submit a single job per cluster. Having two ways to do the same thing is > > confusing. > > * Per-Job Mode is only supported by YARN anyway. If we keep it, we should > > work towards support in Kubernetes and Standalone, too, to reduce special > > casing. > > * Dropping per-job mode would reduce complexity in the code and allow us > to > > dedicate more resources to the other two deployment modes. > > * I believe with session mode and application mode we have to easily > > distinguishable and understandable deployment modes that cover Flink's > use > > cases: > > * session
RE: [DISCUSS] Future of Per-Job Mode
Hi Konstantin, First of all, sorry for the delay. We at Cloudera are currently relying on per-job mode deploying Flink applications over YARN. Specifically, we allow users to upload connector jars and other artifacts. There are also some default jars that we need to ship. These are all stored on the local file system of our service’s node. The Flink job is submitted on the users’ behalf by our service, which also specifies the jars to ship. The service runs on a single node, not on all nodes with Flink TM/JM. It would thus be difficult to manage the jars on every node. We are not familiar with the reasoning behind why application mode currently doesn’t ship the user jars, besides the deployment being faster this way. Would it be possible for the application mode to (optionally, enabled by some config) distribute these, or are there some technical limitations? For us it would be crucial to achieve the functionality we have at the moment over YARN. We started to track https://issues.apache.org/jira/browse/FLINK-24897 that Biao Geng mentioned as well. Considering the above, for us the more soonish removal does not sound really well. We can live with this feature as deprecated of course, but it would be nice to have some time to figure out how we can utilize Application Mode exactly and make necessary changes if required. Thank you, F On 2022/01/13 08:30:48 Konstantin Knauf wrote: > Hi everyone, > > I would like to discuss and understand if the benefits of having Per-Job > Mode in Apache Flink outweigh its drawbacks. > > > *# Background: Flink's Deployment Modes* > Flink currently has three deployment modes. They differ in the following > dimensions: > * main() method executed on Jobmanager or Client > * dependencies shipped by client or bundled with all nodes > * number of jobs per cluster & relationship between job and cluster > lifecycle* (supported resource providers) > > ## Application Mode > * main() method executed on Jobmanager > * dependencies already need to be available on all nodes > * dedicated cluster for all jobs executed from the same main()-method > (Note: applications with more than one job, currently still significant > limitations like missing high-availability). Technically, a session cluster > dedicated to all jobs submitted from the same main() method. > * supported by standalone, native kubernetes, YARN > > ## Session Mode > * main() method executed in client > * dependencies are distributed from and by the client to all nodes > * cluster is shared by multiple jobs submitted from different clients, > independent lifecycle > * supported by standalone, Native Kubernetes, YARN > > ## Per-Job Mode > * main() method executed in client > * dependencies are distributed from and by the client to all nodes > * dedicated cluster for a single job > * supported by YARN only > > > *# Reasons to Keep** There are use cases where you might need the > combination of a single job per cluster, but main() method execution in the > client. This combination is only supported by per-job mode. > * It currently exists. Existing users will need to migrate to either > session or application mode. > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > modes that for most users probably do the same thing. Specifically, for > those users that don't care where the main() method is executed and want to > submit a single job per cluster. Having two ways to do the same thing is > confusing. > * Per-Job Mode is only supported by YARN anyway. If we keep it, we should > work towards support in Kubernetes and Standalone, too, to reduce special > casing. > * Dropping per-job mode would reduce complexity in the code and allow us to > dedicate more resources to the other two deployment modes. > * I believe with session mode and application mode we have to easily > distinguishable and understandable deployment modes that cover Flink's use > cases: > * session mode: olap-style, interactive jobs/queries, short lived batch > jobs, very small jobs, traditional cluster-centric deployment mode (fits > the "Hadoop world") > * application mode: long-running streaming jobs, large scale & > heterogenous jobs (resource isolation!), application-centric deployment > mode (fits the "Kubernetes world") > > > *# Call to Action* > * Do you use per-job mode? If so, why & would you be able to migrate to one > of the other methods? > * Am I missing any pros/cons? > * Are you in favor of dropping per-job mode midterm? > > Cheers and thank you, > > Konstantin > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
Re: [DISCUSS] Future of Per-Job Mode
Hi all, I agree with Xintong's comment: Reducing the number of deployment modes would help users. There is a clearer distinction between session mode and the two other deployment modes (i.e. application and job mode). The difference between application and job mode is not that easy to grasp for newcomers, I imagine. It would also help cleaning up some job-mode-specific code segments in the source code. It would be interesting to see whether there are other use-cases that are missed in the Application mode (besides the ones already addressed by Biao). I would second Xintong's proposal of deprecating the job-mode rather soonish making users aware of the plans around that deployment mode. That might help encourage users to speak up in case they are not able to find a solution to work around deprecation warnings. I also agree with Xintong's assessment that dropping it should only be done after we're sure that all relevant use cases are met also by other deployment modes considering that (based on the comments above) it is a widely used deployment mode. Matthias On Mon, Jan 24, 2022 at 10:00 AM Xintong Song wrote: > Sorry for joining the discussion late. > > I'm leaning towards deprecating the per-job mode soonish, and eventually > dropping it in the long-term. > - One less deployment mode makes it easier for users (especially > newcomers) to understand. Deprecating the per-job mode sends the signal > that it is legacy, not recommended, and in most cases users do not need to > care about it. > - For most (if not all) user demands that are satisfied by the per-job > mode but not by the application mode, AFAICS, they can be either workaround > or eventually addressed by the application mode. E.g., make application > mode support shipping local dependencies. > - I'm not sure about dropping the per-job mode soonish, as many users are > still working with it. We'd better not force these users to migrate to the > application mode when upgrading the Flink version. > > Thank you~ > > Xintong Song > > > > On Fri, Jan 21, 2022 at 4:30 PM Konstantin Knauf > wrote: > >> Thanks Thomas & Biao for your feedback. >> >> Any additional opinions on how we should proceed with per job-mode? As >> you might have guessed, I am leaning towards proposing to deprecate per-job >> mode. >> >> On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote: >> >>> Regarding session mode: >>> >>> ## Session Mode >>> * main() method executed in client >>> >>> Session mode also supports execution of the main method on Jobmanager >>> with submission through REST API. That's how Flinkk k8s operators like >>> [1] work. It's actually an important capability because it allows for >>> allocation of the cluster resources prior to taking down the previous >>> job during upgrade when the goal is optimization for availability. >>> >>> Thanks, >>> Thomas >>> >>> [1] https://github.com/lyft/flinkk8soperator >>> >>> On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf >>> wrote: >>> > >>> > Hi everyone, >>> > >>> > I would like to discuss and understand if the benefits of having >>> Per-Job >>> > Mode in Apache Flink outweigh its drawbacks. >>> > >>> > >>> > *# Background: Flink's Deployment Modes* >>> > Flink currently has three deployment modes. They differ in the >>> following >>> > dimensions: >>> > * main() method executed on Jobmanager or Client >>> > * dependencies shipped by client or bundled with all nodes >>> > * number of jobs per cluster & relationship between job and cluster >>> > lifecycle* (supported resource providers) >>> > >>> > ## Application Mode >>> > * main() method executed on Jobmanager >>> > * dependencies already need to be available on all nodes >>> > * dedicated cluster for all jobs executed from the same main()-method >>> > (Note: applications with more than one job, currently still significant >>> > limitations like missing high-availability). Technically, a session >>> cluster >>> > dedicated to all jobs submitted from the same main() method. >>> > * supported by standalone, native kubernetes, YARN >>> > >>> > ## Session Mode >>> > * main() method executed in client >>> > * dependencies are distributed from and by the client to all nodes >>> > * cluster is shared by multiple jobs submitted from different clients, >>> > independent lifecycle >>> > * supported by standalone, Native Kubernetes, YARN >>> > >>> > ## Per-Job Mode >>> > * main() method executed in client >>> > * dependencies are distributed from and by the client to all nodes >>> > * dedicated cluster for a single job >>> > * supported by YARN only >>> > >>> > >>> > *# Reasons to Keep** There are use cases where you might need the >>> > combination of a single job per cluster, but main() method execution >>> in the >>> > client. This combination is only supported by per-job mode. >>> > * It currently exists. Existing users will need to migrate to either >>> > session or application mode. >>> > >>> > >>> > *# Reasons to Drop** With Per-Job Mode and Application Mode we
Re: [DISCUSS] Future of Per-Job Mode
Sorry for joining the discussion late. I'm leaning towards deprecating the per-job mode soonish, and eventually dropping it in the long-term. - One less deployment mode makes it easier for users (especially newcomers) to understand. Deprecating the per-job mode sends the signal that it is legacy, not recommended, and in most cases users do not need to care about it. - For most (if not all) user demands that are satisfied by the per-job mode but not by the application mode, AFAICS, they can be either workaround or eventually addressed by the application mode. E.g., make application mode support shipping local dependencies. - I'm not sure about dropping the per-job mode soonish, as many users are still working with it. We'd better not force these users to migrate to the application mode when upgrading the Flink version. Thank you~ Xintong Song On Fri, Jan 21, 2022 at 4:30 PM Konstantin Knauf wrote: > Thanks Thomas & Biao for your feedback. > > Any additional opinions on how we should proceed with per job-mode? As you > might have guessed, I am leaning towards proposing to deprecate per-job > mode. > > On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote: > >> Regarding session mode: >> >> ## Session Mode >> * main() method executed in client >> >> Session mode also supports execution of the main method on Jobmanager >> with submission through REST API. That's how Flinkk k8s operators like >> [1] work. It's actually an important capability because it allows for >> allocation of the cluster resources prior to taking down the previous >> job during upgrade when the goal is optimization for availability. >> >> Thanks, >> Thomas >> >> [1] https://github.com/lyft/flinkk8soperator >> >> On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf >> wrote: >> > >> > Hi everyone, >> > >> > I would like to discuss and understand if the benefits of having Per-Job >> > Mode in Apache Flink outweigh its drawbacks. >> > >> > >> > *# Background: Flink's Deployment Modes* >> > Flink currently has three deployment modes. They differ in the following >> > dimensions: >> > * main() method executed on Jobmanager or Client >> > * dependencies shipped by client or bundled with all nodes >> > * number of jobs per cluster & relationship between job and cluster >> > lifecycle* (supported resource providers) >> > >> > ## Application Mode >> > * main() method executed on Jobmanager >> > * dependencies already need to be available on all nodes >> > * dedicated cluster for all jobs executed from the same main()-method >> > (Note: applications with more than one job, currently still significant >> > limitations like missing high-availability). Technically, a session >> cluster >> > dedicated to all jobs submitted from the same main() method. >> > * supported by standalone, native kubernetes, YARN >> > >> > ## Session Mode >> > * main() method executed in client >> > * dependencies are distributed from and by the client to all nodes >> > * cluster is shared by multiple jobs submitted from different clients, >> > independent lifecycle >> > * supported by standalone, Native Kubernetes, YARN >> > >> > ## Per-Job Mode >> > * main() method executed in client >> > * dependencies are distributed from and by the client to all nodes >> > * dedicated cluster for a single job >> > * supported by YARN only >> > >> > >> > *# Reasons to Keep** There are use cases where you might need the >> > combination of a single job per cluster, but main() method execution in >> the >> > client. This combination is only supported by per-job mode. >> > * It currently exists. Existing users will need to migrate to either >> > session or application mode. >> > >> > >> > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two >> > modes that for most users probably do the same thing. Specifically, for >> > those users that don't care where the main() method is executed and >> want to >> > submit a single job per cluster. Having two ways to do the same thing is >> > confusing. >> > * Per-Job Mode is only supported by YARN anyway. If we keep it, we >> should >> > work towards support in Kubernetes and Standalone, too, to reduce >> special >> > casing. >> > * Dropping per-job mode would reduce complexity in the code and allow >> us to >> > dedicate more resources to the other two deployment modes. >> > * I believe with session mode and application mode we have to easily >> > distinguishable and understandable deployment modes that cover Flink's >> use >> > cases: >> >* session mode: olap-style, interactive jobs/queries, short lived >> batch >> > jobs, very small jobs, traditional cluster-centric deployment mode (fits >> > the "Hadoop world") >> >* application mode: long-running streaming jobs, large scale & >> > heterogenous jobs (resource isolation!), application-centric deployment >> > mode (fits the "Kubernetes world") >> > >> > >> > *# Call to Action* >> > * Do you use per-job mode? If so, why & would you be able to migrate to >> one >> > of the
Re: [DISCUSS] Future of Per-Job Mode
Thanks Thomas & Biao for your feedback. Any additional opinions on how we should proceed with per job-mode? As you might have guessed, I am leaning towards proposing to deprecate per-job mode. On Thu, Jan 13, 2022 at 5:11 PM Thomas Weise wrote: > Regarding session mode: > > ## Session Mode > * main() method executed in client > > Session mode also supports execution of the main method on Jobmanager > with submission through REST API. That's how Flinkk k8s operators like > [1] work. It's actually an important capability because it allows for > allocation of the cluster resources prior to taking down the previous > job during upgrade when the goal is optimization for availability. > > Thanks, > Thomas > > [1] https://github.com/lyft/flinkk8soperator > > On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf > wrote: > > > > Hi everyone, > > > > I would like to discuss and understand if the benefits of having Per-Job > > Mode in Apache Flink outweigh its drawbacks. > > > > > > *# Background: Flink's Deployment Modes* > > Flink currently has three deployment modes. They differ in the following > > dimensions: > > * main() method executed on Jobmanager or Client > > * dependencies shipped by client or bundled with all nodes > > * number of jobs per cluster & relationship between job and cluster > > lifecycle* (supported resource providers) > > > > ## Application Mode > > * main() method executed on Jobmanager > > * dependencies already need to be available on all nodes > > * dedicated cluster for all jobs executed from the same main()-method > > (Note: applications with more than one job, currently still significant > > limitations like missing high-availability). Technically, a session > cluster > > dedicated to all jobs submitted from the same main() method. > > * supported by standalone, native kubernetes, YARN > > > > ## Session Mode > > * main() method executed in client > > * dependencies are distributed from and by the client to all nodes > > * cluster is shared by multiple jobs submitted from different clients, > > independent lifecycle > > * supported by standalone, Native Kubernetes, YARN > > > > ## Per-Job Mode > > * main() method executed in client > > * dependencies are distributed from and by the client to all nodes > > * dedicated cluster for a single job > > * supported by YARN only > > > > > > *# Reasons to Keep** There are use cases where you might need the > > combination of a single job per cluster, but main() method execution in > the > > client. This combination is only supported by per-job mode. > > * It currently exists. Existing users will need to migrate to either > > session or application mode. > > > > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > > modes that for most users probably do the same thing. Specifically, for > > those users that don't care where the main() method is executed and want > to > > submit a single job per cluster. Having two ways to do the same thing is > > confusing. > > * Per-Job Mode is only supported by YARN anyway. If we keep it, we should > > work towards support in Kubernetes and Standalone, too, to reduce special > > casing. > > * Dropping per-job mode would reduce complexity in the code and allow us > to > > dedicate more resources to the other two deployment modes. > > * I believe with session mode and application mode we have to easily > > distinguishable and understandable deployment modes that cover Flink's > use > > cases: > >* session mode: olap-style, interactive jobs/queries, short lived > batch > > jobs, very small jobs, traditional cluster-centric deployment mode (fits > > the "Hadoop world") > >* application mode: long-running streaming jobs, large scale & > > heterogenous jobs (resource isolation!), application-centric deployment > > mode (fits the "Kubernetes world") > > > > > > *# Call to Action* > > * Do you use per-job mode? If so, why & would you be able to migrate to > one > > of the other methods? > > * Am I missing any pros/cons? > > * Are you in favor of dropping per-job mode midterm? > > > > Cheers and thank you, > > > > Konstantin > > > > -- > > > > Konstantin Knauf > > > > https://twitter.com/snntrable > > > > https://github.com/knaufk > -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
Re: [DISCUSS] Future of Per-Job Mode
Regarding session mode: ## Session Mode * main() method executed in client Session mode also supports execution of the main method on Jobmanager with submission through REST API. That's how Flinkk k8s operators like [1] work. It's actually an important capability because it allows for allocation of the cluster resources prior to taking down the previous job during upgrade when the goal is optimization for availability. Thanks, Thomas [1] https://github.com/lyft/flinkk8soperator On Thu, Jan 13, 2022 at 12:32 AM Konstantin Knauf wrote: > > Hi everyone, > > I would like to discuss and understand if the benefits of having Per-Job > Mode in Apache Flink outweigh its drawbacks. > > > *# Background: Flink's Deployment Modes* > Flink currently has three deployment modes. They differ in the following > dimensions: > * main() method executed on Jobmanager or Client > * dependencies shipped by client or bundled with all nodes > * number of jobs per cluster & relationship between job and cluster > lifecycle* (supported resource providers) > > ## Application Mode > * main() method executed on Jobmanager > * dependencies already need to be available on all nodes > * dedicated cluster for all jobs executed from the same main()-method > (Note: applications with more than one job, currently still significant > limitations like missing high-availability). Technically, a session cluster > dedicated to all jobs submitted from the same main() method. > * supported by standalone, native kubernetes, YARN > > ## Session Mode > * main() method executed in client > * dependencies are distributed from and by the client to all nodes > * cluster is shared by multiple jobs submitted from different clients, > independent lifecycle > * supported by standalone, Native Kubernetes, YARN > > ## Per-Job Mode > * main() method executed in client > * dependencies are distributed from and by the client to all nodes > * dedicated cluster for a single job > * supported by YARN only > > > *# Reasons to Keep** There are use cases where you might need the > combination of a single job per cluster, but main() method execution in the > client. This combination is only supported by per-job mode. > * It currently exists. Existing users will need to migrate to either > session or application mode. > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > modes that for most users probably do the same thing. Specifically, for > those users that don't care where the main() method is executed and want to > submit a single job per cluster. Having two ways to do the same thing is > confusing. > * Per-Job Mode is only supported by YARN anyway. If we keep it, we should > work towards support in Kubernetes and Standalone, too, to reduce special > casing. > * Dropping per-job mode would reduce complexity in the code and allow us to > dedicate more resources to the other two deployment modes. > * I believe with session mode and application mode we have to easily > distinguishable and understandable deployment modes that cover Flink's use > cases: >* session mode: olap-style, interactive jobs/queries, short lived batch > jobs, very small jobs, traditional cluster-centric deployment mode (fits > the "Hadoop world") >* application mode: long-running streaming jobs, large scale & > heterogenous jobs (resource isolation!), application-centric deployment > mode (fits the "Kubernetes world") > > > *# Call to Action* > * Do you use per-job mode? If so, why & would you be able to migrate to one > of the other methods? > * Am I missing any pros/cons? > * Are you in favor of dropping per-job mode midterm? > > Cheers and thank you, > > Konstantin > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk
Re: [DISCUSS] Future of Per-Job Mode
Hi Konstantin, Thanks a lot for starting this discussion! I hope my thoughts and experiences why users use Per-Job Mode, especially in YARN can help: #1. Per-job mode makes managing dependencies easier: I have met some customers who used Per-Job Mode to submit jobs with a lot of local user-defined jars by using '-C' option directly. They do not need to upload these jars to some remote file system(e.g. HDFS) first, which makes their life easier. #2. In YARN mode, currently, there are some limitations of Application Mode: in this jira(https://issues.apache.org/jira/browse/FLINK-24897) that I am working on, we find that YARN Application Mode do not support `usrlib` very well, which makes it hard to use FlinkUserCodeClassLoader to load classes in user-defined jars. I believe above 2 points, especially #2, can be reassured as we enhance the YARN Application Mode later but I think it is worthwhile to consider dependency management more carefully before we make decisions. Best, Biao Geng Konstantin Knauf 于2022年1月13日周四 16:32写道： > Hi everyone, > > I would like to discuss and understand if the benefits of having Per-Job > Mode in Apache Flink outweigh its drawbacks. > > > *# Background: Flink's Deployment Modes* > Flink currently has three deployment modes. They differ in the following > dimensions: > * main() method executed on Jobmanager or Client > * dependencies shipped by client or bundled with all nodes > * number of jobs per cluster & relationship between job and cluster > lifecycle* (supported resource providers) > > ## Application Mode > * main() method executed on Jobmanager > * dependencies already need to be available on all nodes > * dedicated cluster for all jobs executed from the same main()-method > (Note: applications with more than one job, currently still significant > limitations like missing high-availability). Technically, a session cluster > dedicated to all jobs submitted from the same main() method. > * supported by standalone, native kubernetes, YARN > > ## Session Mode > * main() method executed in client > * dependencies are distributed from and by the client to all nodes > * cluster is shared by multiple jobs submitted from different clients, > independent lifecycle > * supported by standalone, Native Kubernetes, YARN > > ## Per-Job Mode > * main() method executed in client > * dependencies are distributed from and by the client to all nodes > * dedicated cluster for a single job > * supported by YARN only > > > *# Reasons to Keep** There are use cases where you might need the > combination of a single job per cluster, but main() method execution in the > client. This combination is only supported by per-job mode. > * It currently exists. Existing users will need to migrate to either > session or application mode. > > > *# Reasons to Drop** With Per-Job Mode and Application Mode we have two > modes that for most users probably do the same thing. Specifically, for > those users that don't care where the main() method is executed and want to > submit a single job per cluster. Having two ways to do the same thing is > confusing. > * Per-Job Mode is only supported by YARN anyway. If we keep it, we should > work towards support in Kubernetes and Standalone, too, to reduce special > casing. > * Dropping per-job mode would reduce complexity in the code and allow us > to dedicate more resources to the other two deployment modes. > * I believe with session mode and application mode we have to easily > distinguishable and understandable deployment modes that cover Flink's use > cases: >* session mode: olap-style, interactive jobs/queries, short lived batch > jobs, very small jobs, traditional cluster-centric deployment mode (fits > the "Hadoop world") >* application mode: long-running streaming jobs, large scale & > heterogenous jobs (resource isolation!), application-centric deployment > mode (fits the "Kubernetes world") > > > *# Call to Action* > * Do you use per-job mode? If so, why & would you be able to migrate to > one of the other methods? > * Am I missing any pros/cons? > * Are you in favor of dropping per-job mode midterm? > > Cheers and thank you, > > Konstantin > > -- > > Konstantin Knauf > > https://twitter.com/snntrable > > https://github.com/knaufk >
[DISCUSS] Future of Per-Job Mode
Hi everyone, I would like to discuss and understand if the benefits of having Per-Job Mode in Apache Flink outweigh its drawbacks. *# Background: Flink's Deployment Modes* Flink currently has three deployment modes. They differ in the following dimensions: * main() method executed on Jobmanager or Client * dependencies shipped by client or bundled with all nodes * number of jobs per cluster & relationship between job and cluster lifecycle* (supported resource providers) ## Application Mode * main() method executed on Jobmanager * dependencies already need to be available on all nodes * dedicated cluster for all jobs executed from the same main()-method (Note: applications with more than one job, currently still significant limitations like missing high-availability). Technically, a session cluster dedicated to all jobs submitted from the same main() method. * supported by standalone, native kubernetes, YARN ## Session Mode * main() method executed in client * dependencies are distributed from and by the client to all nodes * cluster is shared by multiple jobs submitted from different clients, independent lifecycle * supported by standalone, Native Kubernetes, YARN ## Per-Job Mode * main() method executed in client * dependencies are distributed from and by the client to all nodes * dedicated cluster for a single job * supported by YARN only *# Reasons to Keep** There are use cases where you might need the combination of a single job per cluster, but main() method execution in the client. This combination is only supported by per-job mode. * It currently exists. Existing users will need to migrate to either session or application mode. *# Reasons to Drop** With Per-Job Mode and Application Mode we have two modes that for most users probably do the same thing. Specifically, for those users that don't care where the main() method is executed and want to submit a single job per cluster. Having two ways to do the same thing is confusing. * Per-Job Mode is only supported by YARN anyway. If we keep it, we should work towards support in Kubernetes and Standalone, too, to reduce special casing. * Dropping per-job mode would reduce complexity in the code and allow us to dedicate more resources to the other two deployment modes. * I believe with session mode and application mode we have to easily distinguishable and understandable deployment modes that cover Flink's use cases: * session mode: olap-style, interactive jobs/queries, short lived batch jobs, very small jobs, traditional cluster-centric deployment mode (fits the "Hadoop world") * application mode: long-running streaming jobs, large scale & heterogenous jobs (resource isolation!), application-centric deployment mode (fits the "Kubernetes world") *# Call to Action* * Do you use per-job mode? If so, why & would you be able to migrate to one of the other methods? * Am I missing any pros/cons? * Are you in favor of dropping per-job mode midterm? Cheers and thank you, Konstantin -- Konstantin Knauf https://twitter.com/snntrable https://github.com/knaufk
