Re: Custom scheduler in Flink
thanks. I saw the google docs just now. I am gonna print and study it at the weekend. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Fri, Nov 30, 2018 at 4:42 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > thanks. I will check it out > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > > On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann > wrote: > >> Hi Felipe, >> >> https://issues.apache.org/jira/browse/FLINK-10429 might also be >> interesting. The community is currently working on making the Scheduler >> pluggable to make it easier to extend this component. >> >> Cheers, >> Till >> >> On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez < >> felipe.o.gutier...@gmail.com> wrote: >> >>> Thanks, I'll check it out. >>> *--* >>> *-- Felipe Gutierrez* >>> >>> *-- skype: felipe.o.gutierrez* >>> *--* *https://felipeogutierrez.blogspot.com >>> <https://felipeogutierrez.blogspot.com>* >>> >>> >>> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler >>> wrote: >>> >>> > There's no *reasonable *way to implement a custom Scheduler, i.e., >>> > something where can just plug in your scheduler in a nice way. For this >>> > you'll have to directly modify the source of Flink. >>> > >>> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also >>> be >>> > of interest, but is still in the design phase. >>> > >>> > On 28.11.2018 10:21, Felipe Gutierrez wrote: >>> > >>> > Hi, >>> > >>> > I want to develop a custom scheduler in Flink to be aware for which >>> host >>> > Flink must process some task. This post shows (using Apache Storm) the >>> kind >>> > of example I want to build ( >>> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e >>> > ). >>> > >>> > I saw some related question on this post from 2016 ( >>> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html >>> ) >>> > and I wonder if there is something more actual or better explained >>> about a >>> > custom scheduler. >>> > >>> > Thanks, >>> > Felipe >>> > *--* >>> > *-- Felipe Gutierrez* >>> > >>> > *-- skype: felipe.o.gutierrez* >>> > *--* *https://felipeogutierrez.blogspot.com< >>> https://felipeogutierrez.blogspot.com> < >>> https://felipeogutierrez.blogspot.com>* >>> > >>> > >>> > >>> > >>> >>
Re: Custom scheduler in Flink
thanks. I will check it out *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Fri, Nov 30, 2018 at 3:51 PM Till Rohrmann wrote: > Hi Felipe, > > https://issues.apache.org/jira/browse/FLINK-10429 might also be > interesting. The community is currently working on making the Scheduler > pluggable to make it easier to extend this component. > > Cheers, > Till > > On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez < > felipe.o.gutier...@gmail.com> wrote: > >> Thanks, I'll check it out. >> *--* >> *-- Felipe Gutierrez* >> >> *-- skype: felipe.o.gutierrez* >> *--* *https://felipeogutierrez.blogspot.com >> <https://felipeogutierrez.blogspot.com>* >> >> >> On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler >> wrote: >> >> > There's no *reasonable *way to implement a custom Scheduler, i.e., >> > something where can just plug in your scheduler in a nice way. For this >> > you'll have to directly modify the source of Flink. >> > >> > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also >> be >> > of interest, but is still in the design phase. >> > >> > On 28.11.2018 10:21, Felipe Gutierrez wrote: >> > >> > Hi, >> > >> > I want to develop a custom scheduler in Flink to be aware for which host >> > Flink must process some task. This post shows (using Apache Storm) the >> kind >> > of example I want to build ( >> https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e >> > ). >> > >> > I saw some related question on this post from 2016 ( >> http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html >> ) >> > and I wonder if there is something more actual or better explained >> about a >> > custom scheduler. >> > >> > Thanks, >> > Felipe >> > *--* >> > *-- Felipe Gutierrez* >> > >> > *-- skype: felipe.o.gutierrez* >> > *--* *https://felipeogutierrez.blogspot.com< >> https://felipeogutierrez.blogspot.com> < >> https://felipeogutierrez.blogspot.com>* >> > >> > >> > >> > >> >
Re: Custom scheduler in Flink
Hi Felipe, https://issues.apache.org/jira/browse/FLINK-10429 might also be interesting. The community is currently working on making the Scheduler pluggable to make it easier to extend this component. Cheers, Till On Wed, Nov 28, 2018 at 2:56 PM Felipe Gutierrez < felipe.o.gutier...@gmail.com> wrote: > Thanks, I'll check it out. > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* *https://felipeogutierrez.blogspot.com > <https://felipeogutierrez.blogspot.com>* > > > On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler > wrote: > > > There's no *reasonable *way to implement a custom Scheduler, i.e., > > something where can just plug in your scheduler in a nice way. For this > > you'll have to directly modify the source of Flink. > > > > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be > > of interest, but is still in the design phase. > > > > On 28.11.2018 10:21, Felipe Gutierrez wrote: > > > > Hi, > > > > I want to develop a custom scheduler in Flink to be aware for which host > > Flink must process some task. This post shows (using Apache Storm) the > kind > > of example I want to build ( > https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e > > ). > > > > I saw some related question on this post from 2016 ( > http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html > ) > > and I wonder if there is something more actual or better explained about > a > > custom scheduler. > > > > Thanks, > > Felipe > > *--* > > *-- Felipe Gutierrez* > > > > *-- skype: felipe.o.gutierrez* > > *--* *https://felipeogutierrez.blogspot.com< > https://felipeogutierrez.blogspot.com> < > https://felipeogutierrez.blogspot.com>* > > > > > > > > >
Re: Custom scheduler in Flink
Thanks, I'll check it out. *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>* On Wed, Nov 28, 2018 at 2:44 PM Chesnay Schepler wrote: > There's no *reasonable *way to implement a custom Scheduler, i.e., > something where can just plug in your scheduler in a nice way. For this > you'll have to directly modify the source of Flink. > > The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be > of interest, but is still in the design phase. > > On 28.11.2018 10:21, Felipe Gutierrez wrote: > > Hi, > > I want to develop a custom scheduler in Flink to be aware for which host > Flink must process some task. This post shows (using Apache Storm) the kind > of example I want to build > (https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e > ). > > I saw some related question on this post from 2016 > (http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html) > and I wonder if there is something more actual or better explained about a > custom scheduler. > > Thanks, > Felipe > *--* > *-- Felipe Gutierrez* > > *-- skype: felipe.o.gutierrez* > *--* > *https://felipeogutierrez.blogspot.com<https://felipeogutierrez.blogspot.com> > <https://felipeogutierrez.blogspot.com>* > > > >
Re: Custom scheduler in Flink
There's no /reasonable /way to implement a custom Scheduler, i.e., something where can just plug in your scheduler in a nice way. For this you'll have to directly modify the source of Flink. The work in https://issues.apache.org/jira/browse/FLINK-8886 may also be of interest, but is still in the design phase. On 28.11.2018 10:21, Felipe Gutierrez wrote: Hi, I want to develop a custom scheduler in Flink to be aware for which host Flink must process some task. This post shows (using Apache Storm) the kind of example I want to build ( https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e ). I saw some related question on this post from 2016 ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html) and I wonder if there is something more actual or better explained about a custom scheduler. Thanks, Felipe *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>*
Custom scheduler in Flink
Hi, I want to develop a custom scheduler in Flink to be aware for which host Flink must process some task. This post shows (using Apache Storm) the kind of example I want to build ( https://inside.edited.com/taking-control-of-your-apache-storm-cluster-with-tag-aware-scheduling-b605e37e ). I saw some related question on this post from 2016 ( http://apache-flink-user-mailing-list-archive.2336050.n4.nabble.com/custom-scheduler-in-Flink-td7804.html) and I wonder if there is something more actual or better explained about a custom scheduler. Thanks, Felipe *--* *-- Felipe Gutierrez* *-- skype: felipe.o.gutierrez* *--* *https://felipeogutierrez.blogspot.com <https://felipeogutierrez.blogspot.com>*
Re: custom scheduler in Flink?
Hi, I'm afraid there is no documentation about schedulers, especially at this low level. Maybe this new design proposal would of interest for you, though: https://cwiki.apache.org/confluence/display/FLINK/FLIP-1+%3A+Fine+Grained+Recovery+from+Task+Failures In there is a link to the mailing list discussion and the design doc in Google Docs. Cheers, Aljoscha On Tue, 5 Jul 2016 at 05:12 #ZHANG SHUHAO# <szhang...@e.ntu.edu.sg> wrote: > Hi everyone, > > > > I want to develop a custom task scheduler in Flink, specifically, I want > to control the scheduling of components of an topology into specific hosts, > where should I start with? Is there an example program or tutorial about > this? Thanks! > > > > Thanks. > > Best regards. > > Shuhao Zhang (Tony) > > +65-86166722 > > >
custom scheduler in Flink?
Hi everyone, I want to develop a custom task scheduler in Flink, specifically, I want to control the scheduling of components of an topology into specific hosts, where should I start with? Is there an example program or tutorial about this? Thanks! Thanks. Best regards. Shuhao Zhang (Tony) +65-86166722
