+1 It's already a matter of fact for a while that we no longer port new features to the Mesos deployment.
Thank you~ Xintong Song On Fri, Mar 26, 2021 at 10:37 PM Till Rohrmann <trohrm...@apache.org> wrote: > +1 for officially deprecating this component for the 1.13 release. > > Cheers, > Till > > On Thu, Mar 25, 2021 at 1:49 PM Konstantin Knauf <kna...@apache.org> > wrote: > >> Hi Matthias, >> >> Thank you for following up on this. +1 to officially deprecate Mesos in >> the code and documentation, too. It will be confusing for users if this >> diverges from the roadmap. >> >> Cheers, >> >> Konstantin >> >> On Thu, Mar 25, 2021 at 12:23 PM Matthias Pohl <matth...@ververica.com> >> wrote: >> >>> Hi everyone, >>> considering the upcoming release of Flink 1.13, I wanted to revive the >>> discussion about the Mesos support ones more. Mesos is also already >>> listed >>> as deprecated in Flink's overall roadmap [1]. Maybe, it's time to align >>> the >>> documentation accordingly to make it more explicit? >>> >>> What do you think? >>> >>> Best, >>> Matthias >>> >>> [1] https://flink.apache.org/roadmap.html#feature-radar >>> >>> On Wed, Oct 28, 2020 at 9:40 AM Till Rohrmann <trohrm...@apache.org> >>> wrote: >>> >>> > Hi Oleksandr, >>> > >>> > yes you are right. The biggest problem is at the moment the lack of >>> test >>> > coverage and thereby confidence to make changes. We have some e2e tests >>> > which you can find here [1]. These tests are, however, quite coarse >>> grained >>> > and are missing a lot of cases. One idea would be to add a Mesos e2e >>> test >>> > based on Flink's end-to-end test framework [2]. I think what needs to >>> be >>> > done there is to add a Mesos resource and a way to submit jobs to a >>> Mesos >>> > cluster to write e2e tests. >>> > >>> > [1] https://github.com/apache/flink/tree/master/flink-jepsen >>> > [2] >>> > >>> https://github.com/apache/flink/tree/master/flink-end-to-end-tests/flink-end-to-end-tests-common >>> > >>> > Cheers, >>> > Till >>> > >>> > On Tue, Oct 27, 2020 at 12:29 PM Oleksandr Nitavskyi < >>> > o.nitavs...@criteo.com> wrote: >>> > >>> >> Hello Xintong, >>> >> >>> >> Thanks for the insights and support. >>> >> >>> >> Browsing the Mesos backlog and didn't identify anything critical, >>> which >>> >> is left there. >>> >> >>> >> I see that there are were quite a lot of contributions to the Flink >>> Mesos >>> >> in the recent version: >>> >> https://github.com/apache/flink/commits/master/flink-mesos. >>> >> We plan to validate the current Flink master (or release 1.12 branch) >>> our >>> >> Mesos setup. In case of any issues, we will try to propose changes. >>> >> My feeling is that our test results shouldn't affect the Flink 1.12 >>> >> release cycle. And if any potential commits will land into the 1.12.1 >>> it >>> >> should be totally fine. >>> >> >>> >> In the future, we would be glad to help you guys with any >>> >> maintenance-related questions. One of the highest priorities around >>> this >>> >> component seems to be the development of the full e2e test. >>> >> >>> >> Kind Regards >>> >> Oleksandr Nitavskyi >>> >> ________________________________ >>> >> From: Xintong Song <tonysong...@gmail.com> >>> >> Sent: Tuesday, October 27, 2020 7:14 AM >>> >> To: dev <d...@flink.apache.org>; user <user@flink.apache.org> >>> >> Cc: Piyush Narang <p.nar...@criteo.com> >>> >> Subject: [BULK]Re: [SURVEY] Remove Mesos support >>> >> >>> >> Hi Piyush, >>> >> >>> >> Thanks a lot for sharing the information. It would be a great relief >>> that >>> >> you are good with Flink on Mesos as is. >>> >> >>> >> As for the jira issues, I believe the most essential ones should have >>> >> already been resolved. You may find some remaining open issues here >>> [1], >>> >> but not all of them are necessary if we decide to keep Flink on Mesos >>> as is. >>> >> >>> >> At the moment and in the short future, I think helps are mostly >>> needed on >>> >> testing the upcoming release 1.12 with Mesos use cases. The community >>> is >>> >> currently actively preparing the new release, and hopefully we could >>> come >>> >> up with a release candidate early next month. It would be greatly >>> >> appreciated if you fork as experienced Flink on Mesos users can help >>> with >>> >> verifying the release candidates. >>> >> >>> >> >>> >> Thank you~ >>> >> >>> >> Xintong Song >>> >> >>> >> [1] >>> >> >>> https://issues.apache.org/jira/browse/FLINK-17402?jql=project%20%3D%20FLINK%20AND%20component%20%3D%20%22Deployment%20%2F%20Mesos%22%20AND%20status%20%3D%20Open >>> >> < >>> >> >>> https://eur03.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FFLINK-17402%3Fjql%3Dproject%2520%253D%2520FLINK%2520AND%2520component%2520%253D%2520%2522Deployment%2520%252F%2520Mesos%2522%2520AND%2520status%2520%253D%2520Open&data=04%7C01%7Co.nitavskyi%40criteo.com%7C3585e1f25bdf4e091af808d87a3f92db%7C2a35d8fd574d48e3927c8c398e225a01%7C1%7C0%7C637393760750820881%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwiLCJXVCI6Mn0%3D%7C1000&sdata=hytJFQE0MCPzMLiQTQTdbg3GVckX5M3r1NPRGrRV8j4%3D&reserved=0 >>> >> > >>> >> >>> >> On Tue, Oct 27, 2020 at 2:58 AM Piyush Narang <p.nar...@criteo.com >>> >> <mailto:p.nar...@criteo.com>> wrote: >>> >> >>> >> Hi Xintong, >>> >> >>> >> >>> >> >>> >> Do you have any jiras that cover any of the items on 1 or 2? I can >>> reach >>> >> out to folks internally and see if I can get some folks to commit to >>> >> helping out. >>> >> >>> >> >>> >> >>> >> To cover the other qs: >>> >> >>> >> * Yes, we’ve not got a plan at the moment to get off Mesos. We use >>> >> Yarn for some our Flink workloads when we can. Mesos is only used >>> when we >>> >> need streaming capabilities in our WW dcs (as our Yarn is centralized >>> in >>> >> one DC) >>> >> * We’re currently on Flink 1.9 (old planner). We have a plan to >>> bump >>> >> to 1.11 / 1.12 this quarter. >>> >> * We typically upgrade once every 6 months to a year (not every >>> >> release). We’d like to speed up the cadence but we’re not there yet. >>> >> * We’d largely be good with keeping Flink on Mesos as-is and >>> >> functional while missing out on some of the newer features. We >>> understand >>> >> the pain on the communities side and we can take on the work if we >>> see some >>> >> fancy improvement in Flink on Yarn / K8s that we want in Mesos to put >>> in >>> >> the request to port it over. >>> >> >>> >> >>> >> >>> >> Thanks, >>> >> >>> >> >>> >> >>> >> -- Piyush >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> From: Xintong Song <tonysong...@gmail.com<mailto: >>> tonysong...@gmail.com>> >>> >> Date: Sunday, October 25, 2020 at 10:57 PM >>> >> To: dev <d...@flink.apache.org<mailto:d...@flink.apache.org>>, user < >>> >> user@flink.apache.org<mailto:user@flink.apache.org>> >>> >> Cc: Lasse Nedergaard <lassenedergaardfl...@gmail.com<mailto: >>> >> lassenedergaardfl...@gmail.com>>, <p.nar...@criteo.com<mailto: >>> >> p.nar...@criteo.com>> >>> >> Subject: Re: [SURVEY] Remove Mesos support >>> >> >>> >> >>> >> >>> >> Thanks for sharing the information with us, Piyush an Lasse. >>> >> >>> >> >>> >> >>> >> @Piyush >>> >> >>> >> >>> >> >>> >> Thanks for offering the help. IMO, there are currently several >>> problems >>> >> that make supporting Flink on Mesos challenging for us. >>> >> >>> >> 1. Lack of Mesos experts. AFAIK, there are very few people (if not >>> >> none) among the active contributors in this community that are >>> familiar >>> >> with Mesos and can help with development on this component. >>> >> 2. Absence of tests. Mesos does not provide a testing cluster, like >>> >> `MiniYARNCluster`, making it hard to test interactions between Flink >>> and >>> >> Mesos. We have only a few very simple e2e tests running on Mesos >>> deployed >>> >> in a docker, covering the most fundamental workflows. We are not sure >>> how >>> >> well those tests work, especially against some potential corner cases. >>> >> 3. Divergence from other deployment. Because of 1 and 2, the new >>> >> efforts (features, maintenance, refactors) tend to exclude Mesos if >>> >> possible. When the new efforts have to touch the Mesos related >>> components >>> >> (e.g., changes to the common resource manager interfaces), we have to >>> be >>> >> very careful and make as few changes as possible, to avoid >>> accidentally >>> >> breaking anything that we are not familiar with. As a result, the >>> component >>> >> diverges a lot from other deployment components (K8s/Yarn), which >>> makes it >>> >> harder to maintain. >>> >> >>> >> It would be greatly appreciated if you can help with either of the >>> above >>> >> issues. >>> >> >>> >> >>> >> >>> >> Additionally, I have a few questions concerning your use cases at >>> Criteo. >>> >> IIUC, you are going to stay on Mesos in the foreseeable future, while >>> >> keeping the Flink version up-to-date? What Flink version are you >>> currently >>> >> using? How often do you upgrade (e.g., every release)? Would you be >>> good >>> >> with keeping the Flink on Mesos component as it is (means that >>> deployment >>> >> and resource management improvements may not be ported to Mesos), >>> while >>> >> keeping other components up-to-date (e.g., improvements from >>> programming >>> >> APIs, operators, state backens, etc.)? >>> >> >>> >> >>> >> >>> >> Thank you~ >>> >> >>> >> Xintong Song >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> On Sat, Oct 24, 2020 at 2:48 AM Lasse Nedergaard < >>> >> lassenedergaardfl...@gmail.com<mailto:lassenedergaardfl...@gmail.com >>> >> >>> >> wrote: >>> >> >>> >> Hi >>> >> >>> >> >>> >> >>> >> At Trackunit We have been using Mesos for long time but have now >>> moved to >>> >> k8s. >>> >> >>> >> Med venlig hilsen / Best regards >>> >> >>> >> Lasse Nedergaard >>> >> >>> >> >>> >> >>> >> >>> >> >>> >> Den 23. okt. 2020 kl. 17.01 skrev Robert Metzger <rmetz...@apache.org >>> >> <mailto:rmetz...@apache.org>>: >>> >> >>> >> >>> >> >>> >> Hey Piyush, >>> >> >>> >> thanks a lot for raising this concern. I believe we should keep Mesos >>> in >>> >> Flink then in the foreseeable future. >>> >> >>> >> Your offer to help is much appreciated. We'll let you know once there >>> is >>> >> something. >>> >> >>> >> >>> >> >>> >> On Fri, Oct 23, 2020 at 4:28 PM Piyush Narang <p.nar...@criteo.com >>> >> <mailto:p.nar...@criteo.com>> wrote: >>> >> >>> >> Thanks Kostas. If there's items we can help with, I'm sure we'd be >>> able >>> >> to find folks who would be excited to contribute / help in any way. >>> >> >>> >> -- Piyush >>> >> >>> >> >>> >> On 10/23/20, 10:25 AM, "Kostas Kloudas" <kklou...@gmail.com<mailto: >>> >> kklou...@gmail.com>> wrote: >>> >> >>> >> Thanks Piyush for the message. >>> >> After this, I revoke my +1. I agree with the previous opinions >>> that we >>> >> cannot drop code that is actively used by users, especially if it >>> >> something that deep in the stack as support for cluster management >>> >> framework. >>> >> >>> >> Cheers, >>> >> Kostas >>> >> >>> >> On Fri, Oct 23, 2020 at 4:15 PM Piyush Narang < >>> p.nar...@criteo.com >>> >> <mailto:p.nar...@criteo.com>> wrote: >>> >> > >>> >> > Hi folks, >>> >> > >>> >> > >>> >> > >>> >> > We at Criteo are active users of the Flink on Mesos resource >>> >> management component. We are pretty heavy users of Mesos for >>> scheduling >>> >> workloads on our edge datacenters and we do want to continue to be >>> able to >>> >> run some of our Flink topologies (to compute machine learning short >>> term >>> >> features) on those DCs. If possible our vote would be not to drop >>> Mesos >>> >> support as that will tie us to an old release / have to maintain a >>> fork as >>> >> we’re not planning to migrate off Mesos anytime soon. Is the burden >>> >> something that can be helped with by the community? (Or are you >>> referring >>> >> to having to ensure PRs handle the Mesos piece as well when they >>> touch the >>> >> resource managers?) >>> >> > >>> >> > >>> >> > >>> >> > Thanks, >>> >> > >>> >> > >>> >> > >>> >> > -- Piyush >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > From: Till Rohrmann <trohrm...@apache.org<mailto: >>> >> trohrm...@apache.org>> >>> >> > Date: Friday, October 23, 2020 at 8:19 AM >>> >> > To: Xintong Song <tonysong...@gmail.com<mailto: >>> >> tonysong...@gmail.com>> >>> >> > Cc: dev <d...@flink.apache.org<mailto:d...@flink.apache.org>>, >>> user < >>> >> user@flink.apache.org<mailto:user@flink.apache.org>> >>> >> > Subject: Re: [SURVEY] Remove Mesos support >>> >> > >>> >> > >>> >> > >>> >> > Thanks for starting this survey Robert! I second Konstantin and >>> >> Xintong in the sense that our Mesos user's opinions should matter most >>> >> here. If our community is no longer using the Mesos integration, then >>> I >>> >> would be +1 for removing it in order to decrease the maintenance >>> burden. >>> >> > >>> >> > >>> >> > >>> >> > Cheers, >>> >> > >>> >> > Till >>> >> > >>> >> > >>> >> > >>> >> > On Fri, Oct 23, 2020 at 2:03 PM Xintong Song < >>> tonysong...@gmail.com >>> >> <mailto:tonysong...@gmail.com>> wrote: >>> >> > >>> >> > +1 for adding a warning in 1.12 about planning to remove Mesos >>> >> support. >>> >> > >>> >> > >>> >> > >>> >> > With my developer hat on, removing the Mesos support would >>> >> definitely reduce the maintaining overhead for the deployment and >>> resource >>> >> management related components. On the other hand, the Flink on Mesos >>> users' >>> >> voices definitely matter a lot for this community. Either way, it >>> would be >>> >> good to draw users attention to this discussion early. >>> >> > >>> >> > >>> >> > >>> >> > Thank you~ >>> >> > >>> >> > Xintong Song >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > >>> >> > On Fri, Oct 23, 2020 at 7:53 PM Konstantin Knauf < >>> kna...@apache.org >>> >> <mailto:kna...@apache.org>> wrote: >>> >> > >>> >> > Hi Robert, >>> >> > >>> >> > +1 to the plan you outlined. If we were to drop support in Flink >>> >> 1.13+, we >>> >> > would still support it in Flink 1.12- with bug fixes for some >>> time >>> >> so that >>> >> > users have time to move on. >>> >> > >>> >> > It would certainly be very interesting to hear from current >>> Flink >>> >> on Mesos >>> >> > users, on how they see the evolution of this part of the >>> ecosystem. >>> >> > >>> >> > Best, >>> >> > >>> >> > Konstantin >>> >> >>> > >>> >> >> >> -- >> >> Konstantin Knauf >> >> https://twitter.com/snntrable >> >> https://github.com/knaufk >> >
