Welcome Charles-François Natali as Mesos Committer and PMC Member!
Hi all, Please welcome Charles-François as a committer and PMC member of the Apache Mesos project! Charles started contributing to Mesos a bit more than a year ago. During this time, he has discovered and fixed a significant number of issues in all corners of Mesos, from libprocess and the build system to the cgroups isolator, thus becoming the most active contributor since November 2020. Also, Charles has been helping a lot with JIRA triage, mentoring other contributors and moving forward a number of technical discussions. Charles, thanks for all your contributions so far and looking forward to continuing to work with you on the project! - Andrei
Re: Permissions to close Jira Issues
Looks like the "Contributors" group of the JIRA project should have this permission; I've just added you there manually, now it should work, please try again. I don't remember if there has ever been any automation to fill this group based on 'contributors.yaml'. On Thu, 20 May 2021 at 11:33, Andreas Peters wrote: > > Hi Qian, > > yes thats what I mean. But there is nothing to mark it as resolved. :-( > > Regards, > Andreas > > Am 20.05.21 um 10:22 schrieb Qian Zhang: > > Hi Andreas > > > > Do you mean marking issues in https://issues.apache.org/jira/ as resolved? > > I do not think you need special permission to do that. > > > > > > Regards, > > Qian Zhang > > > > > > On Thu, May 20, 2021 at 3:18 PM Andreas Peters > > wrote: > > > >> Hey, > >> > >> > >> what I have to do, to get the permissions to close issues in "our" Jira? > >> > >> > >> Regards, > >> Andreas > >> > >> > > > > -- > AVENTER UG (haftungsbeschraenkt) > Köllner Chaussee 144 > 25337 Kölln-Reisiek > > Phone: (+49)4121 - 235582 0 > E-Mail: a...@aventer.biz > Internet: https://www.aventer.biz > Git: https://www.github.com/AVENTER-UG > > Geschäftsführer/CEO: Andreas Peters > Unternehmenssitz/City: Kölln-Reisiek > Handelsregister beim Amtsgericht/Legal Court: Itzehoe > Handelsregister-Nummer/Company Registration Number: HRB 9995 PI > > Diese E-Mail kann vertrauliche und/oder rechtlich geschützte > Informationen enthalten. Wenn Sie nicht der beabsichtigte Empfänger > sind oder diese E-Mail irrtümlich erhalten haben, informieren Sie > bitte sofort den Absender telefonisch oder per E-Mail und löschen > Sie diese E-Mail aus Ihrem System. Das unerlaubte Kopieren sowie > die unbefugte Weitergabe dieser Mail ist nicht gestattet. > > > This email can be confidential and/or legally protected > Information included. If you are not the intended recipient > or have received this e-mail in error, inform your > contact and the sender immediately by phone or email and delete this > email from your system. Unauthorized copying as well > the unauthorized forwarding of this mail is not permitted. >
Re: [VOTE] Move Apache Mesos to Attic
Well, one of the strengths of this project used to be effective collaboration between people sitting in different locations and different time zones (and, at certain intervals of time, working in different companies). Talking about COVID... the pandemic, assuming that it will subside, should probably be one more reason to vote -1 at this point in time. I don't know about everyone, but the additional overhead it creates everywhere is, for example, absolutely not helping me spend any time outside my working hours on any kind of knowledge transfer in Mesos (including discussions on the future of the project, for that matter). On Thu, 8 Apr 2021 at 18:29, Andreas Peters wrote: > > As I see, Andrei you are sitting close to me (Hamburg, Germany), thats > just 30km away from me. :-) It would make mentoring more easy, if we can > see each other (of course, after covid). > > So, Andrei provide to be a mentor, I provide to be his student. :-) > > Cheers, > Andreas > >
Re: [VOTE] Move Apache Mesos to Attic
EDIT: I will reconsider my vote if someone explains how the latter (lower the bar and remain in ASF) is MORE harmful to the project than the former (lower the bar and leave ASF) // Sorry, mixing up left and right, as usual On Thu, 8 Apr 2021 at 18:10, Andrei Sekretenko wrote: > > Sorry guys, let me try to fail this vote. > -1 (binding) > > To continue any development (including maintenance), the entry bar for > the people allowed to merge code and participate in project decisions > has to be lowered, this way or another. > > I do not see how the approach "let's move the project out of ASF and > lower the bar there" benefits the _future_ of the project, compared to > lowering the committer bar while remaining in ASF. > (I will reconsider my vote if someone explains how the latter is less > harmful to the project than the former) > > To me, the elimination of the ASF reporting overhead is quite > marginal, compared to the added overhead of: > - potential for future trademark hurdles (which can scare any > potential contribution there might be) > - the need to immediately switch CI and issue tracker > - the old JIRA going read-only (yes, this should be counted as a > separate concern) > and so on > > I'm not saying that Mesos should stick to the ASF-provided > infrastructure forever, but this is a different story. > There are a number of things in Mesos that require major overhaul to > lower the contribution overhead; in the new conditions, this might be > one of them. > > Getting current (and potential future) contributors unstuck and > relieving Vinod from the duties of heading the project (given that he > wants to step down from being the PMC chair) is something we should do > ASAP, however I doubt that moving into Attic is the right way to do > that. > > I do not feel especially optimistic about the future prospects of the > project, but IMO moving to Attic makes them even worse. > > > --- > Talking about what I personally can/want/cannot do: > - I do not see myself actively contributing (=adding new features and > submitting solutions to address technical debt) to the project in the > foreseeable future > - I will not be surprised if I will need to contribute some minor but > critical bugfix in the nearest few months; for example, if someone > exposes a major vulnerability in Mesos, I will probably have to work > on that, and work on that quickly > - I'm ready to provide technical mentoring (and, hopefully, > reasonably quick reviews) in areas I'm familiar with (this, > unfortunately, excludes most of the inner workings of the agent, for > example) > - I can provide mentoring and code reviews in areas I'm less familiar > with, but do not expect quick feedback from me there, as meaningfully > reviewing an unfamiliar code can be 2 to 100 times slower > - Despite having an opinion on where Mesos should not go, some > understanding of the project's past mistakes, and some ideas on what > it should drop, I have absolutely no idea where Mesos should go. > > Best, > Andrei > > On Thu, 8 Apr 2021 at 17:50, Benjamin Bannier wrote: > > > > Hi Shane, > > > > >> FWIW, I'm one of those people who said they were interested, and I > > >> still voted to move it to the attic (even though my vote is non > > >> binding as I'm not a committer). > > > > > >That's great! One of the questions I have for the project is: why > > >haven't they made you a committer yet? (That's a question for the PMC, > > >not for you, really, but it's one I'm betting the Board would be curious > > >to hear about). > > > > I fear the problem is less that contributors haven't been voted in as > > committers, but IMO that unfortunately the project has seen hardly any > > contributions of bigger patches from the community in the last couple of > > years, (I know we merged patches from Charles, but when some people > > voiced interested here that's the first at least I heard of them). This > > makes me pessimistic regarding a handover which medium term can improve > > the situation. > > > > I summarized some issues I saw in an earlier thread (yes, this discussion > > has been going on since Feb 2021), and I think the issues were only if > > at all marginally related to committership, > > https://lists.apache.org/thread.html/r4bccbf048a9bcde3f0bb66d5e2c57f585296e1f5e2769486413b2758%40%3Cdev.mesos.apache.org%3E > > . > > > > > > Cheers, > > > > Benjamin
Re: [VOTE] Move Apache Mesos to Attic
Sorry guys, let me try to fail this vote. -1 (binding) To continue any development (including maintenance), the entry bar for the people allowed to merge code and participate in project decisions has to be lowered, this way or another. I do not see how the approach "let's move the project out of ASF and lower the bar there" benefits the _future_ of the project, compared to lowering the committer bar while remaining in ASF. (I will reconsider my vote if someone explains how the latter is less harmful to the project than the former) To me, the elimination of the ASF reporting overhead is quite marginal, compared to the added overhead of: - potential for future trademark hurdles (which can scare any potential contribution there might be) - the need to immediately switch CI and issue tracker - the old JIRA going read-only (yes, this should be counted as a separate concern) and so on I'm not saying that Mesos should stick to the ASF-provided infrastructure forever, but this is a different story. There are a number of things in Mesos that require major overhaul to lower the contribution overhead; in the new conditions, this might be one of them. Getting current (and potential future) contributors unstuck and relieving Vinod from the duties of heading the project (given that he wants to step down from being the PMC chair) is something we should do ASAP, however I doubt that moving into Attic is the right way to do that. I do not feel especially optimistic about the future prospects of the project, but IMO moving to Attic makes them even worse. --- Talking about what I personally can/want/cannot do: - I do not see myself actively contributing (=adding new features and submitting solutions to address technical debt) to the project in the foreseeable future - I will not be surprised if I will need to contribute some minor but critical bugfix in the nearest few months; for example, if someone exposes a major vulnerability in Mesos, I will probably have to work on that, and work on that quickly - I'm ready to provide technical mentoring (and, hopefully, reasonably quick reviews) in areas I'm familiar with (this, unfortunately, excludes most of the inner workings of the agent, for example) - I can provide mentoring and code reviews in areas I'm less familiar with, but do not expect quick feedback from me there, as meaningfully reviewing an unfamiliar code can be 2 to 100 times slower - Despite having an opinion on where Mesos should not go, some understanding of the project's past mistakes, and some ideas on what it should drop, I have absolutely no idea where Mesos should go. Best, Andrei On Thu, 8 Apr 2021 at 17:50, Benjamin Bannier wrote: > > Hi Shane, > > >> FWIW, I'm one of those people who said they were interested, and I > >> still voted to move it to the attic (even though my vote is non > >> binding as I'm not a committer). > > > >That's great! One of the questions I have for the project is: why > >haven't they made you a committer yet? (That's a question for the PMC, > >not for you, really, but it's one I'm betting the Board would be curious > >to hear about). > > I fear the problem is less that contributors haven't been voted in as > committers, but IMO that unfortunately the project has seen hardly any > contributions of bigger patches from the community in the last couple of > years, (I know we merged patches from Charles, but when some people > voiced interested here that's the first at least I heard of them). This > makes me pessimistic regarding a handover which medium term can improve > the situation. > > I summarized some issues I saw in an earlier thread (yes, this discussion > has been going on since Feb 2021), and I think the issues were only if > at all marginally related to committership, > https://lists.apache.org/thread.html/r4bccbf048a9bcde3f0bb66d5e2c57f585296e1f5e2769486413b2758%40%3Cdev.mesos.apache.org%3E > . > > > Cheers, > > Benjamin
Future technical direction of Mesos?
Hi all, while the vote on moving to Attic is ongoing (I hope that the current standstill will end quite soon - by Attic+non-ASF fork, by lowering the committer bar or in some other way), I think we need to (once again) ask recent and potential future contributors and users: In which exact direction would you like to see the project moving? What are the things you need in Mesos and would be willing to contribute? For the people and organizations who are maintaining private Mesos forks - what is there in those forks that might be of value to the community and you would like to merge into upstream? I'm asking, because I'm under impression that an (unconscious) view of some/many of the PMC is that Mesos is not - and will never be - solving any task that Kubernetes is not solving now and will not be solving in the foreseeable future. _If_ one views Mesos+something as a "contender in a container orchestration war" then that "battle" is clearly lost. Using the "bad analogy" Curl/Firefox by Charles-François Natali: what are the tasks for which you use this "Curl" where it cannot be reasonably substituted by "Firefox"? And do you envision other tasks of these kinds? -- In the other words: what do you want Mesos to be in the future? A thing capable of running legacy Mesos frameworks for container orchestration? A core of a system which struggles to keep feature parity with solutions on top of K8s, has performance characteristics similar to those solutions, but is not built on top of K8s? Or... something entirely different? In the two first cases, I would agree that Mesos has no future and should be retired (and retired not only as an ASF project, but in principle). In the latter case - can you be more specific? Note that, although the answers might impact the ongoing Attic vote, the project needs them regardless of the outcome of that vote. Best regards, Andrei Sekretenko
Re: Next Steps
IIUC, Attic is not intended for projects which still have active users and thus might be in need of fixing bugs. Key items about moving project to Attic: > It is not intended to: > - Rebuild community > - Make bugfixes > - Make releases >Projects whose PMC are unable to muster 3 votes for a release, who have no >active committers or are unable to fulfill their reporting duties to the board >are all good candidates for the Attic. As a D2iQ employee, I can say that if we find a bug critical for our customers, we will be interested in fixing that. Should the project be moved into Attic, the fix will be present only in forks (which might mean our internal forks). I could imagine that other entities and people using Mesos are in a similar position with regards to bugfixes. If this is true, then moving the project to Attic in the near future is not a proper solution to the issue of insufficient bandwidth of the active PMC members/chair. --- A long-term future of the project is a different story, which, in my personal view, will "end" either in moving the project into Attic or in shifting the project direction from what it used to be in the recent few years to something substantially different. IMO, this requires a _separate_ discussion. Damien's questions sound like a good starting point for that discussion, I'll try to answer them from my committer/PMC member perspective when I have enough time. On Thu, 18 Feb 2021 at 12:49, Charles-François Natali wrote: > > Thanks Tomek, that's what I suspected. > It would therefore make it much more difficult for anyone to carry on since > it would effectively have to be a fork, etc. > I think it'd be a bit of a shame, but I understand Benjamin's point. > I hope it can be avoided. > > > Cheers, > > > > On Thu, 18 Feb 2021, 11:02 Tomek Janiszewski, wrote: >> >> Moving to attic is making project read only >> https://attic.apache.org/ >> https://attic.apache.org/projects/aurora.html >> >> czw., 18 lut 2021, 11:56 użytkownik Charles-François Natali >> napisał: >>> >>> I'm not familiar with the attic but would it still allow to actually >>> develop, make commits to the repository etc? >>> >>> >>> On Thu, 18 Feb 2021, 08:27 Benjamin Bannier, wrote: >>> >>> > Hi Vinod, >>> > >>> > > I would like to start a discussion around the future of the Mesos >>> > project. >>> > > >>> > > As you are probably aware, the number of active committers and >>> > contributors >>> > > to the project have declined significantly over time. As of today, >>> > there's >>> > > no active development of any features or a public release planned. On >>> > > the >>> > > flip side, I do know there are a few companies who are still actively >>> > using >>> > > Mesos. >>> > >>> > Thanks for starting this discussion Vinod. Looking at Slack, mailing >>> > lists, JIRA and reviewboard/github the project has wound down a lot in >>> > the last 12+ months. >>> > >>> > > Given that, we need to assess if there's interest in the community to >>> > keep >>> > > this project moving forward. Specifically, we need some active >>> > > committers >>> > > and PMC members who are going to manage the project. Ideally, these >>> > > would >>> > > be people who are using Mesos in some capacity and can make code >>> > > contributions. >>> > >>> > While I have seen a few non-committer folks contribute patches in the >>> > last months, I feel it might be too late to bootstrap an active >>> > community at this point. >>> > >>> > Apache Mesos is still mentioned prominently in the docs of a number of >>> > other projects which gives off the impression of an active and >>> > maintained project. In reality almost nobody is working on issues or >>> > available to help users, and basing a new project on Apache Mesos these >>> > days is probably not a good idea. I honestly do not see that to change >>> > should new people step up and IMO the most honest way forward would be >>> > to move the project to the attic to clearly communicate that the project >>> > has moved into another phase; this wouldn't preclude folks from using or >>> > further developing Apache Mesos, but would give a clear signal to users. >>> > >>> > > If there is no active interest, we will likely need to figure out steps >>> > for >>> > > retiring the project. >>> > > >>> > > *Call for action: If you are interested in becoming a committer/PMC >>> > member >>> > > (including PMC chair) and actively maintain the project, please reply to >>> > > this email.* >>> > >>> > Like I wrote above, I would be in favor of a vote to move Apache Mesos >>> > to the attic. >>> > >>> > >>> > Cheers, >>> > >>> > Benjamin >>> >
Subject: [RESULT][VOTE] Release Apache Mesos 1.11.0 (rc1)
Hi all, The vote for Mesos 1.11.0 (rc1) has passed with the following votes. +1 (Binding) -- Vinod Kone Till Toenshoff Qian Zhang Andrei Sekretenko There were no 0 or -1 votes. Please find the release at: https://dist.apache.org/repos/dist/release/mesos/1.11.0 It is recommended to use a mirror to download the release: http://www.apache.org/dyn/closer.cgi The CHANGELOG for the release is available at: https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.11.0 The mesos-1.11.0.jar has been released to: https://repository.apache.org The website (http://mesos.apache.org) will be updated shortly to reflect this release. Thanks, Andrei Sekretenko
Re: Subject: [VOTE] Release Apache Mesos 1.11.0 (rc1)
+1 (binding). Built, ran tests and benchmarked on Ubuntu 18.05 with gcc 7.5.0 (also built and ran binaries/tests with clang 12.0.0). Regards, Andrei Sekretenko On 2020/11/19 01:32:28, Qian Zhang wrote: > +1 > > Regards, > Qian Zhang > > > On Thu, Nov 19, 2020 at 4:16 AM Till Toenshoff wrote: > > > +1 > > > > Build using Apple clang version 12.0.0 (clang-1200.0.32.27) and ran on > > macOS 11.0.1 (20B29). > > > > > On 17. Nov 2020, at 15:53, Andrei Sekretenko > > wrote: > > > > > > Hi all, > > > > > > Please vote on releasing the following candidate as Apache Mesos 1.11.0. > > > > > > 1.11.0 includes the following: > > > > > > > > * CSI external volumes support: now, Mesos Containerizer supports using > > >pre-provisioned external CSI storage volumes by means of the new > > > `volume/csi` > > >isolator. Also, the latter significantly extends the range of > > > compatible 3rd party > > > CSI plugins compared to the previous SLRP-based solution > > > (MESOS-10141). > > > > > > * Constraints-based offer filtering: the Scheduler API adds an > > > interface allowing > > >frameworks to put constraints on agent attributes in resource > > > offers to help "picky" > > >frameworks significantly reduce scheduling latency when close to > > > being out of quota > > >(MESOS-10161). > > > > > > * CMake build becomes usable for deploying in production (MESOS-898). > > > > > > The CHANGELOG for the release is available at: > > > > > https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.11.0-rc1 > > > > > > > > > > > The candidate for Mesos 1.11.0 release is available at: > > > > > https://dist.apache.org/repos/dist/dev/mesos/1.11.0-rc1/mesos-1.11.0.tar.gz > > > > > > The tag to be voted on is 1.11.0-rc1: > > > https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.11.0-rc1 > > > > > > The SHA512 checksum of the tarball can be found at: > > > > > https://dist.apache.org/repos/dist/dev/mesos/1.11.0-rc1/mesos-1.11.0.tar.gz.sha512 > > > > > > The signature of the tarball can be found at: > > > > > https://dist.apache.org/repos/dist/dev/mesos/1.11.0-rc1/mesos-1.11.0.tar.gz.asc > > > > > > The PGP key used to sign the release is here: > > > https://dist.apache.org/repos/dist/release/mesos/KEYS > > > > > > The JAR is in a staging repository here: > > > https://repository.apache.org/content/repositories/orgapachemesos-1260 > > > > > > Please vote on releasing this package as Apache Mesos 1.11.0! > > > > > > The vote is open until 2020 Nov 20th 15:00 UTC at least, and passes if > > > a majority of at least 3 +1 PMC votes are cast. > > > > > > [ ] +1 Release this package as Apache Mesos 1.11.0 > > > [ ] -1 Do not release this package because ... > > > > > > Thanks, > > > Andrei Sekretenko > > > > >
Subject: [VOTE] Release Apache Mesos 1.11.0 (rc1)
Hi all, Please vote on releasing the following candidate as Apache Mesos 1.11.0. 1.11.0 includes the following: * CSI external volumes support: now, Mesos Containerizer supports using pre-provisioned external CSI storage volumes by means of the new `volume/csi` isolator. Also, the latter significantly extends the range of compatible 3rd party CSI plugins compared to the previous SLRP-based solution (MESOS-10141). * Constraints-based offer filtering: the Scheduler API adds an interface allowing frameworks to put constraints on agent attributes in resource offers to help "picky" frameworks significantly reduce scheduling latency when close to being out of quota (MESOS-10161). * CMake build becomes usable for deploying in production (MESOS-898). The CHANGELOG for the release is available at: https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.11.0-rc1 The candidate for Mesos 1.11.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/1.11.0-rc1/mesos-1.11.0.tar.gz The tag to be voted on is 1.11.0-rc1: https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.11.0-rc1 The SHA512 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/1.11.0-rc1/mesos-1.11.0.tar.gz.sha512 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/1.11.0-rc1/mesos-1.11.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1260 Please vote on releasing this package as Apache Mesos 1.11.0! The vote is open until 2020 Nov 20th 15:00 UTC at least, and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 1.11.0 [ ] -1 Do not release this package because ... Thanks, Andrei Sekretenko
Re: Hung website job
Hi Zoran, sure, this particular job can be cancelled/retried. >From looking at the log, there seems to be some infrastructure issue on the ASF Jenkins / Git plugin/system/... level: the build has yet to reach the build script, and looks stuck at the git checkout by the SCM plugin. This might be worth investigating on the infra side (especially given that Jenkins hadn't aborted the stuck job in 300 minutes; I'm not 100% sure if Jenkins is supposed to abort a stuck job at the SCM checkout step, though). On Tue, Oct 27, 2020 at 1:32 PM Zoran Regvart wrote: > Hi Mesos folk, > perhaps someone from your side can take a look? > > thanks! > > zoran > -- Forwarded message - > From: Zoran Regvart > Date: Tue, Oct 27, 2020 at 11:49 AM > Subject: Hung website job > To: builds > > > Hi Builders, > seems that a job has hung[1] on the website1 node preventing other > website jobs to schedule. > > Can it be canceled/retried? > > zoran > [1] https://ci-builds.apache.org/job/Mesos/job/Mesos-Websitebot/716/ > -- > Zoran Regvart > > > -- > Zoran Regvart > -- -- Andrei Sekretenko
Design document: constraints-based offer filtering
Hi all, Recently, I and my colleagues have been designing a mechanism in Mesos that will allow a framework to put constraints on the contents of the offers it receives: on the attributes of offered agents, and, as a next step, on resources in the offers, so that the framework is more likely to receive an offer it really needs. The primary aim of this design is to help "picky" frameworks running in presence of quota reduce scheduling latency. I've distilled the implementation proposal on the Mesos side into a design doc draft: https://docs.google.com/document/d/1MV048BwjLSoa8sn_5hs4kIH4YJMf6-Gsqbij3YuT1No <https://docs.google.com/document/d/1MV048BwjLSoa8sn_5hs4kIH4YJMf6-Gsqbij3YuT1No/edit#heading=h.wq9atl6k4yq0> /edit#heading=h.wq9atl6k4yq0 <https://docs.google.com/document/d/1MV048BwjLSoa8sn_5hs4kIH4YJMf6-Gsqbij3YuT1No/edit#heading=h.wq9atl6k4yq0> -- Best regards, Andrei Sekretenko
Subject: [RESULT][VOTE] Release Apache Mesos 1.10.0 (rc1)
Hi all, The vote for Mesos 1.10.0 (rc1) has passed with the following votes. +1 (Binding) -- Vinod Kone Benjamin Mahler Qian Zhang Greg Mann There were no 0 or -1 votes. Please find the release at: https://dist.apache.org/repos/dist/release/mesos/1.10.0 It is recommended to use a mirror to download the release: http://www.apache.org/dyn/closer.cgi The CHANGELOG for the release is available at: https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.10.0 The mesos-1.10.0.jar has been released to: https://repository.apache.org The website (http://mesos.apache.org) will be updated shortly to reflect this release. Thanks, Andrei Sekretenko
Re: Subject: [VOTE] Release Apache Mesos 1.10.0 (rc1)
Thanks for checking this! The first one (centos, non-SSL, gcc, autotools) seems to be a race between several instances of `javah` attempting to check for existence and create the output directory. I believe there were no related changes in 1.10.x compared to 1.9.x. The second one (ubuntu, SSL, clang, autotools) is somewhat tricky. The immediate cause of the failure seems to be an attempt to compile src/tests/http_tests.proto with a not yet built protoc. src/tests/http_tests.proto has been added in 1.10; there were no tests-only protobuf definitions in Mesos before that. However, I'm not quite getting how protobuf compilation in the automake build is supposed to work at all with a bundled protoc. When the bundled protobuf is used, I don't see any dependency on protoc injected into the pb.cc/pb.h targets in the generated Makefile. Neither do I see how src/Makefile.am is supposed to introduce this dependency. (See https://github.com/apache/mesos/blob/5a04a1693e4f1d51007c23728f1884a307e22src/testssrc/tests9a1/src/Makefile.am#L499 <https://github.com/apache/mesos/blob/5a04a1693e4f1d51007c23728f1884a307e229a1/src/Makefile.am#L499> and below). Looks like all other protobufs (usually?) compile due to sheer luck. The workaround for the javah race and the fix for missing dependency on protoc seem to be rather straightforward. If any of these two should be considered a blocker for 1.10.0, please vote -1. On Tue, May 19, 2020 at 6:55 PM Vinod Kone wrote: > Ran it in Apache CI. Found 2 build issues (issue 1 > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu-eu2)/console>, > issue 2 > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=autotools,COMPILER=clang,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu-eu2)/console>) > which seem to be related to race condition due to parallel build. > > @Andrei Sekretenko Can you confirm this is > not a regression in the build system? > > *Revision*: 1fb36dcc5a0099f147cd01bd82cd7b4f0aec2256 > >- refs/tags/1.10.0-rc1 > > Configuration Matrix gcc clang > centos:7 --verbose --disable-libtool-wrappers > --disable-parallel-test-execution --enable-libevent --enable-ssl autotools > [image: Success] > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu-eu2)/> > [image: Not run] > cmake > [image: Success] > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu-eu2)/> > [image: Not run] > --verbose --disable-libtool-wrappers --disable-parallel-test-execution > autotools > [image: Failed] > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu-eu2)/> > [image: Not run] > cmake > [image: Success] > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=cmake,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=centos%3A7,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu-eu2)/> > [image: Not run] > ubuntu:16.04 --verbose --disable-libtool-wrappers > --disable-parallel-test-execution --enable-libevent --enable-ssl autotools > [image: Success] > <https://builds.apache.org/view/M-R/view/Mesos/job/Mesos-Release/77/BUILDTOOL=autotools,COMPILER=gcc,CONFIGURATION=--verbose%20--disable-libtool-wrappers%20--disable-parallel-test-execution%20--enable-libevent%20--enable-ssl,ENVIRONMENT=GLOG_v=1%20MESOS_VERBOSE=1,OS=ubuntu%3A16.04,label_exp=(docker%7C%7CHadoop%7C%7Cbeam)&&(!ubuntu-us1)&&(!ubuntu
Subject: [VOTE] Release Apache Mesos 1.10.0 (rc1)
Hi all, Please vote on releasing the following candidate as Apache Mesos 1.10.0. 1.10.0 includes the following major improvements: * support for resource bursting (setting task resource limits separately from requests) on Linux * ability for an executor to communicate with an agent via Unix domain socket instead of TCP * ability for operators to modify reservations via the RESERVE_RESOURCES master API call * performance improvements of V1 operator API read-only calls bringing them on par with V0 HTTP endpoints * ability for a scheduler to expect that effects of calls sent through the same connection will not be reordered/interleaved by master NOTE: 1.10.0 includes a breaking change for custom authorizer modules. Now, `ObjectApprover`s may be stored by Mesos indefinitely and must be kept up-to-date by an authorizer throughout their lifetime. This allowed for several bugfixes and performance improvements. The CHANGELOG for the release is available at: https://gitbox.apache.org/repos/asf?p=mesos.git;a=blob_plain;f=CHANGELOG;hb=1.10.0-rc1 The candidate for Mesos 1.10.0 release is available at: https://dist.apache.org/repos/dist/dev/mesos/1.10.0-rc1/mesos-1.10.0.tar.gz The tag to be voted on is 1.10.0-rc1: https://gitbox.apache.org/repos/asf?p=mesos.git;a=commit;h=1.10.0-rc1 The SHA512 checksum of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/1.10.0-rc1/mesos-1.10.0.tar.gz.sha512 The signature of the tarball can be found at: https://dist.apache.org/repos/dist/dev/mesos/1.10.0-rc1/mesos-1.10.0.tar.gz.asc The PGP key used to sign the release is here: https://dist.apache.org/repos/dist/release/mesos/KEYS The JAR is in a staging repository here: https://repository.apache.org/content/repositories/orgapachemesos-1259 Please vote on releasing this package as Apache Mesos 1.10.0! The vote is open until Fri, May 21, 19:00 CEST and passes if a majority of at least 3 +1 PMC votes are cast. [ ] +1 Release this package as Apache Mesos 1.10.0 [ ] -1 Do not release this package because ... Thanks, Andrei Sekretenko
Re: 1.10 release is nearing - please check `Target Version`s in JIRA
Hi, sorry, I probably should have kept things more transparent, here are the recent updates: - 1.10.x was branched off yesterday. All the work that lands on master from now on goes into 1.11 unless backported into 1.10. The preliminary 1.10 changelog can be seen here: https://github.com/apache/mesos/blob/1.10.x/CHANGELOG - I'm wrapping up the final checks before tagging 1.10.0-rc1 release candidate and initiating a release vote (hope that nothing comes up in the remaining checks). On Thu, May 7, 2020 at 10:28 PM Charles-François Natali wrote: > Hey, > > Sorry if I missed something, but I didn't see any follow up on the release > - is there an ETA? > > Cheers, > > > On Fri, 20 Mar 2020, 22:57 Andrei Sekretenko, > wrote: > > > Hi all, > > as some of you probably know, Mesos 1.10 release is near; expect that > next > > week the release candidate will be prepared and voted. > > > > At this point, please make sure that the work that needs to be landed > into > > 1.10 release but has not landed in master branch yet, is labelled by > > TargetVersion = 1.10.0 in the corresponding ASF JIRA ticket. > > --- > > There seem to be unresolved issues in JIRA that had recent activity, but > > are not labelled with TargetVersion = 1.10.0. > > It might be the case that some of them actually need to get to 1.10, but > > are effectively orphaned. > > > > Manually browsing through all the project in a search for these "1.10 > > orphans", so I attempted to filter two lists of issues that have an > > increased probability of containing the orphans: > > https://jira.apache.org/jira/issues/?filter=12348429 > > https://jira.apache.org/jira/issues/?filter=12348426 > > > > Most of these issues do not look like 1.10 blockers to me, but if you > find > > among them (or elsewhere!) some that absolutely need to block 1.10 > release, > > please label them with a proper TargetVersion. > > > > Best, > > Andrei Sekretenko > > > -- -- Andrei Sekretenko
1.10 release is nearing - please check `Target Version`s in JIRA
Hi all, as some of you probably know, Mesos 1.10 release is near; expect that next week the release candidate will be prepared and voted. At this point, please make sure that the work that needs to be landed into 1.10 release but has not landed in master branch yet, is labelled by TargetVersion = 1.10.0 in the corresponding ASF JIRA ticket. --- There seem to be unresolved issues in JIRA that had recent activity, but are not labelled with TargetVersion = 1.10.0. It might be the case that some of them actually need to get to 1.10, but are effectively orphaned. Manually browsing through all the project in a search for these "1.10 orphans", so I attempted to filter two lists of issues that have an increased probability of containing the orphans: https://jira.apache.org/jira/issues/?filter=12348429 https://jira.apache.org/jira/issues/?filter=12348426 Most of these issues do not look like 1.10 blockers to me, but if you find among them (or elsewhere!) some that absolutely need to block 1.10 release, please label them with a proper TargetVersion. Best, Andrei Sekretenko
Design proposal: synchronous authorization in Mesos
Hi all, Probably you are aware that authorization in Mesos, as currently implemented, always requires an asynchronous step: after sending authorization request to Authorizer (or requesting a short-lived ObjectApprover from Authorizer), the caller has to wait (in one way or another) for completion of a Future. Recently, myself and a few of my colleagues started to explore an option to implement synchronous authorization, which will allow to improve Operator event API throughput, to simplify Master code and, most importantly, to get rid of several bugs that are otherwise hard to fix cleanly. The design proposal is here: https://docs.google.com/document/d/12IVdCY668SGJTLhfEWciZfy2EreM_SJ-nzsTb33Tp44/edit?usp=sharing Please take a look and comment! Thanks, Andrei Sekretenko
[custom Authorizers] changing ObjectApprover life cycle inside of Mesos
Hi all, as you probably know, asynchronous authorization step in Mesos master and agent code causes a number of performance/design issues and outright bugs (see https://issues.apache.org/jira/browse/MESOS-10056). To deal with those issues, myself and a few of my colleagues are considering a design which will reuse the existing ObjectApprover interface for synchronous authorization. The main drawback of this design is that existing custom Authorizer implementations will need to be adapted. Currently, Mesos code uses the returned Approver for authorizing a single API call/event, and disposes of the approver immediately after that. Note that Authorizer interface documentation does not require Mesos to behave this way; it states nothing about the lifetime/expiration of the returned ObjectApprover inside of Mesos. The reference implementation (LocalAuthorizer) does not rely on this behaviour, but custom implementations might be implicitly expecting it (some of them indeed are). Using ObjectApprover interface for synchronous authorization will change the ObjectApprover life cycle in Mesos. Mesos master, and probably also agent, will be storing ObjectApprovers for long-lived principals for at least as long as the principal is subscribed. Thus, custom Authorizers will be required to implement a method that returns an ObjectApprover which never expires, with its state refreshed by the Authorizer when needed, in a thread-safe way. We intend to put part of the code needed for approver refresh alongside Mesos, both to facilitate adding Mesos tests with non-idempotent approvers and to simplify adapting custom authorizers. A more detailed design proposal will follow, but before that we would like to figure out: how many custom Authorizer implementations there actually are and how hard will it be for their maintainers to adapt them for this breaking change? If you own (or are aware of) an existing actively used custom Authorizer implementation, please let us know. Thanks, Andrei Sekretenko Software Engineer at D2iQ (previously known as Mesosphere)
Changing behaviour of suppressOffers() to preserve suppressed state on transparent re-registration by the scheduler driver
Hi all,
we are intending to change the behavior of the suppressOffers() method of
MesosSchedulerDriver with regard to the transparent re-registration.
Currently, when driver becomes disconnected from a master, it performs on
its own a re-registration with an empty set of suppressed roles. This
causes un-suppression
of all the suppressed roles of the framework.
The plan is to alter this behavior into preserving the suppression state on
this re-registration.
The required set of suppressed roles will be stored in the driver, which
will be now performing re-registration with this set (instead of an empty
one),
and updating the stored set whenever a call modifying the suppression state
of the roles in the allocator is performed.
Currently, the driver has two methods which perform such calls:
suppressOffers() and reviveOffers().
Please feel free to raise any concerns or objections - especially if you
are aware of any V0 frameworks which (probably implicitly) depend on
un-suppression of the roles when this re-registration occurs.
Note that:
- Frameworks which do not call suppressOffers() are, obviously, unaffected
by this change.
- Frameworks that reliably prevent transparent-re-registration (for
example, by calling driver.abort() immediately from the disconnected()
callback), should also be not affected.
- Storing the suppressed roles list for re-registration and clearing it in
reviveOffers() do not change anything for the existing frameworks. It is
setting this list in suppressOffers() which might be a cause of concerns.
- I'm using the word "un-suppression" because re-registering with roles
removed from the suppressed roles list is NOT equivalent to performing
REVIVE call for these roles (unlike REVIVE, it does not clear offerFilters
in the allocator).
=
A bit of background on why this change is needed.
To properly support V0 frameworks with large number of roles, it is
necessary for the driver not to change the suppression state of the roles
on its own.
Therefore, due to the existence of the transparent re-registration in the
driver, we will need to store the required suppression state in the driver
and make it re-register using this state.
We could possibly avoid the proposed change of suppressOffers() by adding
to the driver new interface for changing the suppression state, leaving
suppressOffers() as it is, and marking it as deprecated.
However, this will leave the behaviour of suppressOffers() deeply
inconsistent with everything else.
Compare the following two sequences of events.
First one:
- The framework creates and starts a driver with roles "role1", "role2"...
"role500", the driver registers
- The framework calls a new method driver.suppressOffersForRoles({"role1",
..., "role500"}), the driver performs SUPPRESS call for these roles and
stores them in its suppressed roles set.
(Alternative with the same result: the framework calls
driver.updateFramework(FrameworkInfo, suppressedRoles={"role1", ...,
"role500"}), the driver performs UPDATE_FRAMEWORK call with those
parameters and stores the new suppressed roles set).
- The driver, due to some reason, disconnects and re-registers with the
same master, providing the stored suppressed roles set.
- All the roles are still suppressed
Second one:
- The framework creates and starts a driver with roles "role1", "role2"...
"role500", the driver registers
- The framework calls driver.suppressOffers(), the driver performs
SUPPRESS call for all roles, but doesn't modify required suppression state.
- The driver, due to some reason, disconnects and re-registers with the
same master, providing the stored suppressed roles set, which is empty.
- Now, none of the roles are suppressed, allocator generates offers for
500 roles which will likely be declined by the framework.
This is one of the examples which makes us strongly consider altering the
interaction between suppressOffers() and the transparent re-registration
when we add storing the suppression state to the driver.
Regards,
Andrei Sekretenko
Bundled glog update from 0.3.3 to 0.4.0
Hi all, We are intending to update the bundled glog from 0.3.3 to 0.4.0. If you have any objections/concerns, or know about any issues introduced into glog between 0.3.3 and 0.4.0, please raise them. Corresponding glog changelogs: https://github.com/google/glog/releases/tag/v0.4.0 https://github.com/google/glog/releases/tag/v0.3.5 https://github.com/google/glog/releases/tag/v0.3.4 Regards, Andrei Sekretenko
