Hi MaGuoWei, If I understand correctly, you're are looking for a way to have a job manager (master) contentiously running which requests YARN containers for the task managers (workers) on the fly. This is currently not supported by Flink although you can add or remove task managers while the cluster is running. The job manager distributes task to the task manager which are available at scheduling time.
As Aljoscha mentioned, the most flexible way of sharing resources in a YARN environment is to start a per-job cluster for each job. Cheers, Max On Mon, Jul 27, 2015 at 11:19 AM, MaGuoWei <maguo...@outlook.com> wrote: > It is great! Is there any document? I am very interested in this. > thanks > > > > From: aljos...@apache.org > > Date: Mon, 27 Jul 2015 05:14:00 +0000 > > Subject: Re: add some new api to the scheduler in the job manager > > To: dev@flink.apache.org > > > > Hi, > > I think for more details on giving back resources of a running cluster we > > have to wait for Robert's opinion. In the mean time, you can also just > run > > a single job that will bring up some yarn containers and then release > them > > afterward using this: > > > https://ci.apache.org/projects/flink/flink-docs-release-0.9/setup/yarn_setup.html#run-a-single-flink-job-on-hadoop-yarn > > > > Cheers, > > Aljoscha > > > > On Mon, 27 Jul 2015 at 03:51 MaGuoWei <maguo...@outlook.com> wrote: > > > > > hi guysFlink must take over all the resources all the time. That wastes > > > resources sometime especially in the sharing cluster.For example when > > > using Flink on Yarn the resource can't be returned back to RM even if > no > > > graph is running.So I want to know: Is it possible to add some apis in > the > > > scheduler to request the resource(slot) and release the resource(slot)? > > > These apis can be customized in the different circumstance. > > > thanks a lot! > > > > > > > > > > >
