Re: [DISCUSS] Spark-Kernel Incubator Proposal
One question about this from the Spark side: have you considered giving the project a different name so that it doesn't sound like a Spark component? Right now "Spark Kernel" may be confused with "Spark Core" and things like that. I don't see a lot of Apache TLPs with related names, though maybe there's nothing wrong with that. In terms of whether to put this in Apache Spark proper, we can have a discussion about it later, but my feeling is that it's not necessary. One reason is that this only uses public APIs, and another is that there are also other notebook interfaces over Spark (e.g. Zeppelin). Matei > On Nov 12, 2015, at 7:17 PM, da...@fallside.com wrote: > > Hello, we would like to start a discussion on accepting the Spark-Kernel, > a mechanism for applications to interactively and remotely access Apache > Spark, into the Apache Incubator. > > The proposal is available online at > https://wiki.apache.org/incubator/SparkKernelProposal, and it is appended > to this email. > > We are looking for additional mentors to help with this project, and we > would much appreciate your guidance and advice. > > Thank-you in advance, > David Fallside > > > > = Spark-Kernel Proposal = > > == Abstract == > Spark-Kernel provides applications with a mechanism to interactively and > remotely access Apache Spark. > > == Proposal == > The Spark-Kernel enables interactive applications to access Apache Spark > clusters. More specifically: > * Applications can send code-snippets and libraries for execution by Spark > * Applications can be deployed separately from Spark clusters and > communicate with the Spark-Kernel using the provided Spark-Kernel client > * Execution results and streaming data can be sent back to calling > applications > * Applications no longer have to be network connected to the workers on a > Spark cluster because the Spark-Kernel acts as each applications proxy > * Work has started on enabling Spark-Kernel to support languages in > addition to Scala, namely Python (with PySpark), R (with SparkR), and SQL > (with SparkSQL) > > == Background & Rationale == > Apache Spark provides applications with a fast and general purpose > distributed computing engine that supports static and streaming data, > tabular and graph representations of data, and an extensive library of > machine learning libraries. Consequently, a wide variety of applications > will be written for Spark and there will be interactive applications that > require relatively frequent function evaluations, and batch-oriented > applications that require one-shot or only occasional evaluation. > > Apache Spark provides two mechanisms for applications to connect with > Spark. The primary mechanism launches applications on Spark clusters using > spark-submit > (http://spark.apache.org/docs/latest/submitting-applications.html); this > requires developers to bundle their application code plus any dependencies > into JAR files, and then submit them to Spark. A second mechanism is an > ODBC/JDBC API > (http://spark.apache.org/docs/latest/sql-programming-guide.html#distributed-sql-engine) > which enables applications to issue SQL queries against SparkSQL. > > Our experience when developing interactive applications, such as analytic > applications and Jupyter Notebooks, to run against Spark was that the > spark-submit mechanism was overly cumbersome and slow (requiring JAR > creation and forking processes to run spark-submit), and the SQL interface > was too limiting and did not offer easy access to components other than > SparkSQL, such as streaming. The most promising mechanism provided by > Apache Spark was the command-line shell > (http://spark.apache.org/docs/latest/programming-guide.html#using-the-shell) > which enabled us to execute code snippets and dynamically control the > tasks submitted to a Spark cluster. Spark does not provide the > command-line shell as a consumable service but it provided us with the > starting point from which we developed the Spark-Kernel. > > == Current Status == > Spark-Kernel was first developed by a small team working on an > internal-IBM Spark-related project in July 2014. In recognition of its > likely general utility to Spark users and developers, in November 2014 the > Spark-Kernel project was moved to GitHub and made available under the > Apache License V2. > > == Meritocracy == > The current developers are familiar with the meritocratic open source > development process at Apache. As the project has gathered interest at > GitHub the developers have actively started a process to invite additional > developers into the project, and we have at least one new developer who is > ready to contribute code to the project. > > == Community == > We started building a community around the Spark-Kernel project when we > moved it to GitHub about one year ago. Since then we have grown to about > 70 people, and there are regular requests and suggestions from the > community. We believe that
Re: [VOTE] Graduation of Apache Spark from the Incubator
+1 On Feb 10, 2014, at 8:27 PM, Chris Mattmann mattm...@apache.org wrote: Hi Everyone, This is a new VOTE to decide if Apache Spark should graduate from the Incubator. Please VOTE on the resolution pasted below the ballot. I'll leave this VOTE open for at least 72 hours. Thanks! [ ] +1 Graduate Apache Spark from the Incubator. [ ] +0 Don't care. [ ] -1 Don't graduate Apache Spark from the Incubator because.. Here is my +1 binding for graduation. Cheers, Chris snip WHEREAS, the Board of Directors deems it to be in the best interests of the Foundation and consistent with the Foundation's purpose to establish a Project Management Committee charged with the creation and maintenance of open-source software, for distribution at no charge to the public, related to fast and flexible large-scale data analysis on clusters. NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to be known as the Apache Spark Project, be and hereby is established pursuant to Bylaws of the Foundation; and be it further RESOLVED, that the Apache Spark Project be and hereby is responsible for the creation and maintenance of software related to fast and flexible large-scale data analysis on clusters; and be it further RESOLVED, that the office of Vice President, Apache Spark be and hereby is created, the person holding such office to serve at the direction of the Board of Directors as the chair of the Apache Spark Project, and to have primary responsibility for management of the projects within the scope of responsibility of the Apache Spark Project; and be it further RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Spark Project: * Mosharaf Chowdhury mosha...@apache.org * Jason Dai jason...@apache.org * Tathagata Das t...@apache.org * Ankur Dave ankurd...@apache.org * Aaron Davidson a...@apache.org * Thomas Dudziak to...@apache.org * Robert Evans bo...@apache.org * Thomas Graves tgra...@apache.org * Andy Konwinski and...@apache.org * Stephen Haberman steph...@apache.org * Mark Hamstra markhams...@apache.org * Shane Huang shane_hu...@apache.org * Ryan LeCompte ryanlecom...@apache.org * Haoyuan Li haoy...@apache.org * Sean McNamara mcnam...@apache.org * Mridul Muralidharam mridul...@apache.org * Kay Ousterhout kayousterh...@apache.org * Nick Pentreath mln...@apache.org * Imran Rashid iras...@apache.org * Charles Reiss wog...@apache.org * Josh Rosen joshro...@apache.org * Prashant Sharma prash...@apache.org * Ram Sriharsha har...@apache.org * Shivaram Venkataraman shiva...@apache.org * Patrick Wendell pwend...@apache.org * Andrew Xia xiajunl...@apache.org * Reynold Xin r...@apache.org * Matei Zaharia ma...@apache.org NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matei Zaharia be appointed to the office of Vice President, Apache Spark, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed; and be it further RESOLVED, that the Apache Spark Project be and hereby is tasked with the migration and rationalization of the Apache Incubator Spark podling; and be it further RESOLVED, that all responsibilities pertaining to the Apache Incubator Spark podling encumbered upon the Apache Incubator Project are hereafter discharged. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduation of Apache Spark from the Incubator
Thanks everyone for the votes. Since we now have accounts for everyone, I’ve updated the committer list below to include ASF IDs. Thanks to INFRA, we also now have both pull request postings and comments on them forwarded to the our mailing list. snip WHEREAS, the Board of Directors deems it to be in the best interests of the Foundation and consistent with the Foundation's purpose to establish a Project Management Committee charged with the creation and maintenance of open-source software, for distribution at no charge to the public, related to fast and flexible large-scale data analysis on clusters. NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to be known as the Apache Spark Project, be and hereby is established pursuant to Bylaws of the Foundation; and be it further RESOLVED, that the Apache Spark Project be and hereby is responsible for the creation and maintenance of software related to efficient cluster management, resource isolation and sharing across distributed applications; and be it further RESOLVED, that the office of Vice President, Apache Spark be and hereby is created, the person holding such office to serve at the direction of the Board of Directors as the chair of the Apache Spark Project, and to have primary responsibility for management of the projects within the scope of responsibility of the Apache Spark Project; and be it further RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Spark Project: * Mosharaf Chowdhury mosha...@apache.org * Jason Dai jason...@apache.org * Tathagata Das t...@apache.org * Ankur Dave ankurd...@apache.org * Aaron Davidson a...@apache.org * Thomas Dudziak to...@apache.org * Robert Evans bo...@apache.org * Thomas Graves tgra...@apache.org * Andy Konwinski and...@apache.org * Stephen Haberman steph...@apache.org * Mark Hamstra markhams...@apache.org * Shane Huang shane_hu...@apache.org * Ryan LeCompte ryanlecom...@apache.org * Haoyuan Li haoy...@apache.org * Sean McNamara mcnam...@apache.org * Mridul Muralidharam mridul...@apache.org * Kay Ousterhout kayousterh...@apache.org * Nick Pentreath mln...@apache.org * Imran Rashid iras...@apache.org * Charles Reiss wog...@apache.org * Josh Rosen joshro...@apache.org * Prashant Sharma prash...@apache.org * Ram Sriharsha har...@apache.org * Shivaram Venkataraman shiva...@apache.org * Patrick Wendell pwend...@apache.org * Andrew Xia xiajunl...@apache.org * Reynold Xin r...@apache.org * Matei Zaharia ma...@apache.org NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matei Zaharia be appointed to the office of Vice President, Apache Spark, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed; and be it further RESOLVED, that the Apache Spark Project be and hereby is tasked with the migration and rationalization of the Apache Incubator Spark podling; and be it further RESOLVED, that all responsibilities pertaining to the Apache Incubator Spark podling encumbered upon the Apache Incubator Project are hereafter discharged. On Feb 8, 2014, at 9:02 AM, David Nalley da...@gnsa.us wrote: Ted and Henry: Thanks for the thoughtful replies and indulging my concerns. +1 (binding) --David On Fri, Feb 7, 2014 at 12:33 AM, Henry Saputra henry.sapu...@gmail.com wrote: HI David Nalley, Thank you for your comment and concern, really appreciate it. As Patrick had mentioned in his reply, this is not a persistent problem. The reminder I sent was about particular topic which could be interpreted as design or roadmap topic rather than review for a patch. Rather than reminding an individual or two involved in the discussion, I decided to send email to dev@ list to show by example the open and transparent discussions the ASF way. Hope this gives some more clarification about the state of the podling embracing the ASF way. - Henry On Thu, Feb 6, 2014 at 9:19 PM, Patrick Wendell pwend...@gmail.com wrote: Hey All - chiming in as an active Spark committer. The part that disturbs me is that after the vote passed in the community, and came to the IPMC a mentor is still having to remind folks that things like strategy and roadmap discussions need to happen on the mailing list. That's a pretty foundational concept in my mind for an Apache project. Henry gave a reminder on the mailing list not because it's a persistent problem but because it never explicitly came up prior to this. We use github for review comments and in one case this week there was a brief discussion that could be interpreted as roadmap - so Henry just gave a reminder not to do that. I can't imagine why any project would *want* to use github review comments for long term roadmap discussion... it's a terrible medium for that anyways! We have a very
Re: Github pull request hooks
Thanks for the very quick action on this, Daniel and Jake! Matei On Feb 8, 2014, at 3:51 AM, Daniel Gruno rum...@cord.dk wrote: Let me just get some information across to podlings wondering about this new GitHub stuff we've added, and what is supported so far: - It is now possible to relay new GitHub PRs, comments on PRs and closing/reopening of PRs to your mailing list. - It is also possible for all committers in a project to close PRs with a commit message to the master/trunk branch of a project saying, for instance, this closes #1234 on GitHub in the commit message. - This GitHub feature is currently only supported for projects/podlings that have their primary source repository in Git. If you have an SVN mirror instead, we are not fully able to support GitHub PR relaying at the time being, but will support it in the near future. - You have to request PR comments to be enabled for your project, preferably via a JIRA ticket (at https://issues.apache.org/jira/browse/INFRA ). By default, hooks have been set up so that only PR opening/closing gets passed on. New setups will include relaying of comments, but if your project has been set up for a web hook in the past, you must request it be set up with comment notification. This is a new default setting, and all projects that now request PR notifications will automatically receive comments on PRs in their respective mailing list as well. If you are unsure whether your project is set up for relaying GitHub messages (and you want this), make a new JIRA ticket to have it set up, rather one ticket too many than one too few. - Comments _should_ now be nested, so that each new comment has an email reference to the original PR issue, meaning they will nest in mod_mbox and your email client if you have nesting/threading enabled. - We may, in the future, add the possibility to filter out certain comments from being relayed to the ML (such as jenkins workflows etc), but this will all depend on how this initial phase goes along. With regards, Daniel. On 02/07/2014 10:02 PM, Daniel Gruno wrote: We had a bit of a snafu on adding PR titles to the subject lines, but it's been taken care of now. With regards, Daniel. On 02/07/2014 09:57 PM, Henry Saputra wrote: Ah looks like the missing PR title only happen for the emails describing comments from the Github pull requests. - Henry On Fri, Feb 7, 2014 at 12:55 PM, Henry Saputra henry.sapu...@gmail.com wrote: Hi Jake, Looks like all the emails from the PR has the same subject: [GitHub] incubator-spark pull request: Is there a way to get the Title of the pull request included in the Subject line? Thanks, Henry On Fri, Feb 7, 2014 at 8:30 AM, Jake Farrell jfarr...@apache.org wrote: I just wanted to follow up on my previous comments about the Github webhooks not posting comments, Daniel Gruno and I have been debugging the Github webhooks we had in place to send pull request notifications and we have fixed it so comments for issues and pull requests will now go to the dev@ lists. If anyone replies to this mail it will stay on the dev@ list. -Jake NOTE: This still remains a project/PMC responsibility to ensure communication is recorded as this is coming from a 3rd party we do not control. Communications should be encouraged to occur on our mailing lists. We (infra) are working to make things easier and allow for better integrations and are always open to anyone wanting to help contribute to these efforts. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduation of Apache Spark from the Incubator
Thanks for describing this, Andrew. We’ll do it as well. Matei On Feb 6, 2014, at 1:24 PM, Andrew Phillips demob...@yahoo.co.uk wrote: The other option would be to somehow subscribe dev@ through the standard GitHub watcher framework by making it the notification address jclouds does that for the jclouds-mirror user (which is the one that periodically mirrors the ASF Git repos which are the source of truth to GitHub). See http://mail-archives.apache.org/mod_mbox/jclouds-notifications/201402.mbox/browser We recently decided to send these emails to notifications@ rather than dev@, because, like Matei, all committers are already watching the GitHub repos and end up getting lots of duplicate emails. Subscribers to the dev@ list were also complaining about the signal-to-noise ratio. [1, 2] ap [1] http://markmail.org/message/4s7spzsiyk45upce [2] https://issues.apache.org/jira/browse/INFRA-7213 - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduation of Apache Spark from the Incubator
I like the idea of sending these to the dev list. I’m going to open an INFRA ticket for it. While it seems that the current web hook only sends events when a pull request is opened or closed, GitHub *can* email watchers for comments as well. I’m watching the project on GitHub and I get an email for every comment, even on pull requests I never looked at. So maybe we can investigate adding this to the GitHub hook — I’m sure it’s not a big burden from their side. The other option would be to somehow subscribe dev@ through the standard GitHub watcher framework by making it the notification address for some account watching the repo, but this requires confirming the address and I think anyone on dev might be able to unsubscribe it or take ownership of that account if we do so. Maybe GitHub can help us do this without going through the “confirm address” process. Did you guys have a contact over there helping with the mirroring, or was it all done through their API? I imagine it’s in their interest to make GitHub work with the ASF process because there are now multiple TLPs using it as their main reviewing system. Matei On Feb 6, 2014, at 10:13 AM, Jake Farrell jfarr...@apache.org wrote: Once the mirror is setup and synced to github (24hr window for this potentially to occur) then one of the Github Apache org admins can setup the webhook. The project just needs to put in an infra ticket asking for it -Jake On Thu, Feb 6, 2014 at 1:06 PM, Henry Saputra henry.sapu...@gmail.comwrote: HI Marvin, The automatic email to dev@ list for github mirror PR, does it happen for all ASF github mirrors or each podling need to do some setup to make it work? - Henry On Thu, Feb 6, 2014 at 9:53 AM, Marvin Humphrey mar...@rectangular.com wrote: On Thu, Feb 6, 2014 at 9:08 AM, Michael Joyce jo...@apache.org wrote: How is doing a review on Github any different than doing a review on Review Board? One is captured to Apache controlled channels and the other is not. If there's concern that work on Github isn't being adequately mirrored on the mailing lists then that sounds like an Infra problem to me. It's not an Infra problem. It's the problem of any PMC which fails to ensure that all of its communications are properly archived. If anyone here is interested in contributing towards this feature, I encourage you to subscribe to the infrastructure-dev@apache list. I also encourage everyone to ponder carefully: * How to ensure that no information is lost when capturing communications in GitHub channels through notifications to our dev lists. * The impact of adding GitHub integration features on long-term ASF Infra labor costs. There are plenty of hooks [1] that makes this easy to do and, if I'm not mistaken, pull requests through Github are already supposed to mail dev@ . That's right. I've actually worked on the specific hook that does that[1][2]. I would assume that any comment on them should as well. Why would you assume that GitHub comments are being mailed to dev lists? They are not. Every Apache PMC member is tasked with oversight of their project, and that includes ensuring that all decisions happen on the dev list and are properly documented. It is not enough to assume that Infra is taking care of archival -- ensuring that the archival actually happens is the PMC's direct responsibility. Marvin Humphrey [1] https://issues.apache.org/jira/browse/INFRA-4651 [2] Thread on legal-discuss@apache: http://s.apache.org/Nhx - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduation of Apache Spark from the Incubator
Hi Craig, Thanks for the list, I’m following up with these folks to get them accounts. I think some people filed an ICLA but never received an account and were thus never added to the repo. A couple of questions: - What do you mean by “does not appear to be a committer” — that they weren’t added to the repo? All of these individuals have contributed code, but it was merged by someone else. - Andrew Xia is listed as having an ICLA on file here: http://people.apache.org/committer-index.html Matei On Feb 4, 2014, at 8:12 AM, Craig L Russell craig.russ...@oracle.com wrote: -1 for this resolution as written. Some of the proposed PMC members have filed ICLAs but are not committers; some have not even filed ICLAs; some appear to have been part of the original proposal but have not contributed to the project during incubation. Regards, Craig On Jan 31, 2014, at 6:04 PM, Matei Zaharia wrote: RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Spark Project: * Mosharaf Chowdhury mosha...@apache.org * Jason Dai jason...@apache.org * Tathagata Das t...@eecs.berkeley.edu This person does not appear to be a committer. * Ankur Dave ankurd...@gmail.com This person does not appear to be a committer. * Aaron Davidson aarondavid...@berkeley.edu This person does not appear to be a committer. * Thomas Dudziak to...@apache.org * Robert Evans bo...@apache.org * Thomas Graves tgra...@apache.org * Andy Konwinski and...@apache.org * Stephen Haberman steph...@apache.org * Mark Hamstra markhams...@apache.org * Shane Huang shane_hu...@apache.org * Ryan LeCompte ryanlecom...@apache.org * Haoyuan Li haoy...@apache.org * Sean McNamara mcnam...@apache.org * Mridul Muralidharam mrid...@yahoo-inc.com This person does not appear to be a committer. * Kay Ousterhout k...@eecs.berkeley.edu There is no ICLA on file for this person. * Nick Pentreath mln...@apache.org * Imran Rashid im...@quantifind.com This person does not appear to be a committer. * Charles Reiss wog...@apache.org * Josh Rosen joshro...@apache.org * Prashant Sharma prash...@apache.org * Ram Sriharsha harsh...@yahoo-inc.com There is no ICLA on file for this person. * Shivaram Venkataraman shiva...@apache.org * Patrick Wendell pwend...@apache.org * Andrew Xia xiajunl...@gmail.com There is no ICLA on file for this person. * Reynold Xin r...@apache.org * Matei Zaharia ma...@apache.org Craig L Russell Architect, Oracle http://db.apache.org/jdo 408 276-5638 mailto:craig.russ...@oracle.com P.S. A good JDO? O, Gasp! - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Graduation of Apache Spark from the Incubator
Hi Craig, On Feb 5, 2014, at 9:05 AM, Craig L Russell craig.russ...@oracle.com wrote: Hi Craig, Thanks for the list, I’m following up with these folks to get them accounts. I think some people filed an ICLA but never received an account and were thus never added to the repo. This is a significant failure of the leadership of this project to request accounts. The project status on http://incubator.apache.org/projects/spark.html says that all active committers have submitted a contributors agreement as of a week ago. The project started seven months ago. Setting up the project, filing ICLAs, and getting accounts for committers is supposed to be part of the initial activities, not a graduation exercise. There was definitely a failure here in setting up their expectations and following up to make sure everyone got an account. All the committers who sent an ICLA requested an account name, but I at least wasn’t clear what the process is for getting one (I assumed that secretary@ does that, since as a non-IPMC member, I can’t create accounts). We also didn’t tell the new committers to ask if they haven’t received an account in X time, so because people were immediately added on the private list they might’ve thought everything is under way. But we can definitely take a much more proactive stance on this as a TLP. I would both 1) tell them to expect an account within a week and 2) make sure the PMC leads this process for new committers. As you must’ve seen from our discussion with secretary@, all the proposed committers actually had submitted ICLAs, and the only one you hadn’t received one from was Kay Ousterhout due to spam filtering. She was only added in December. If you look at the VOTE threads or GitHub review activity, all the added committers were highly active members in terms of both new contributions and reviewing before being proposed as committers. A couple of questions: - What do you mean by “does not appear to be a committer” — that they weren’t added to the repo? They were not given credentials to commit to the repo. All of these individuals have contributed code, but it was merged by someone else. This is a major issue. At Apache, committers update the repo with their own code. Occasionally, they commit code on behalf of others but this should be a rare exception, such as a person from the outside contributing a patch or two. If committers on the project are routinely committing patches on behalf of other active members of the project, there is something fundamentally wrong with the leadership of the project. What I meant by this is that all code is reviewed by another committer and merged by them. Different projects operate differently, but I believe this is a very normal way to operate. I’ve been a committer on Apache Hadoop, one of the most active Apache projects, since 2009, and nearly all the patches I sent there were reviewed and merged by someone else. If you look at the GitHub code reviews (https://github.com/apache/incubator-spark/pulls), you’ll see that lots of people are contributing to reviewing. But I agree that the new committer onboarding process should include having them do a test commit. Perhaps it is a feature of using git that it's so easy to write code, create a pull request, and have someone else do the easy job of merging. That might definitely be part of it. GitHub not only makes it easy to send patches but also makes it easy for the reviewer to say “this patch does not merge cleanly”, or “does not pass unit tests”, so a lot of the merging happens at review time. The actual merging is a couple of shell commands. You can look at some of the pull requests if you’re worried about peoples’ activity — discussion is extremely active and many of our proposed committers were reviewing patches before being proposed. - Andrew Xia is listed as having an ICLA on file here: http://people.apache.org/committer-index.html Yes, I missed this. Andrew's public name is different. There are some folks on the proposed PMC list who do not appear to have been active on the mail lists, which are the life blood of a project. It’s true that some members have been less active in the past six months, but keep in mind that the project existed for 3.5 years before joining the Incubator, and was highly active even then. All the initial committers made major contributions during these first 3.5 years, and as Chris said, our philosophy was to recognize them for their contributions and give them the ability to participate in the project once it moved to Apache. All of the initial committers agreed to being made committers (I asked them before the proposal). If we end up with a problem of inactive committers in the future, we may consider some kind of emeritus status, but I’d personally feel uncomfortable asking anyone to drop their committer status due to the past six months when they’ve
Re: [VOTE] Graduation of Apache Spark from the Incubator
What I meant by this is that all code is reviewed by another committer and merged by them. Different projects operate differently, but I believe this is a very normal way to operate. I’ve been a committer on Apache Hadoop, one of the most active Apache projects, since 2009, and nearly all the patches I sent there were reviewed and merged by someone else. If you look at the GitHub code reviews (https://github.com/apache/incubator-spark/pulls), you’ll see that lots of people are contributing to reviewing. But I agree that the new committer onboarding process should include having them do a test commit. BTW just to reiterate Henry’s point here, because it’s probably not clear coming from SVN: in Git, when you merge someone else’s patch, they retain authorship for it in the commit history. Just do a git log in our repository (e.g. https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=summary). So it’s not a case of “email your patch to someone else and have them apply it”. Matei - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: Incubator ApacheCon presentations
This is an interesting point that some people asked us about (not sure how much it really mattered, but perhaps it did). Maybe the incubator can be re-branded somehow as an “inbound” location or something like that. The name incubator does makes it seem like the software is not mature, but the incubator is of course used for both new software and software being transferred into the ASF. Matei On Feb 2, 2014, at 8:36 AM, Henry Saputra henry.sapu...@gmail.com wrote: I think one of the topics to be covered is about quality. Perception in incubator that it is a low quality so some podlings want to graduate ASAP without learning fully the ropes of being ASF community. - Henry On Fri, Jan 31, 2014 at 11:46 AM, Suresh Marru sma...@apache.org wrote: These are great topics Marvin, looking foreword for them. Hi All, I am planning to propose a talk targeted for recently graduated incubator podlings. Along with sharing my own experiences, I want to highlight importance of giving back to IPMC by hanging around and helping future podlings, and most importantly some of the pitfalls to avoid once we move into self-governance mode. Any volunteers to join in co-presenting this talk or contributing to the content? We don’t need to overwhelm the mailing list and choose to directly email me at smarru at a.o Thanks, Suresh On Jan 29, 2014, at 8:04 PM, Marvin Humphrey mar...@rectangular.com wrote: Greets, I'm planning to propose two presentations for ApacheCon Denver: one on releases, and one overview of the Incubator. _Releasing Apache Software_ This presentation will explore technical, procedural, legal and cultural aspects of releasing Apache software; the distinction between releasing and distributing; the relationship between Apache's source releases and binary artifacts; evolving consensus as to what liberties may be taken with incubating releases; and how codifying the release process helps to fortify the the Apache brand. Audience: Apache committers Level: intermediate _Getting the Most Out of the Incubator_ Knowing what to expect from the incubation process is of great benefit for both contributors to projects currently under incubation and those considering bringing a project to Apache. We'll cover the major stages of incubation, the institutions and roles that incubating projects must interact with, and how to get the most out for what you put in. Audience: Anyone with an interest in the Incubator. Level: beginner Thoughts, feedback? Anybody else submitting talks? Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[VOTE] Graduation of Apache Spark from the Incubator
Hi folks, The Apache Spark community has VOTEd to graduate from the Apache incubator (vote thread: http://s.apache.org/kq, discussion thread: http://s.apache.org/aEQ). I’m now calling an official IPMC VOTE to make this happen as well. Here’s the community tally: +1 Matei Zaharia Reynold Xin Tathagata Das Sean McNamara Patrick Wendell Mark Hamstra Chris Mattmann * Tom Graves Henry Saputra * Andy Konwinski Josh Rosen Mosharaf Chowdhury Mridul Muralidharan Nick Pentreath Andrew Xia Haoyuan Li Sandy Ryza Sebastian Schelter * Kostas Sakellis Christopher Nguyen Aaron Davidson Shivaram Venkataraman Kay Ousterhout Evan Sparks Xuefeng Wu Konstantin Boudnik Rahul Chugh Prashant Sharma Stephen Haberman Prabeesh K. Saisai Shao Junfeng Feng Jason Dai Stevo Slavic Heiko Braun Xia Zhu Manoj Awasthi +0 (none) -1 (none) * indicates IPMC member We’ve made three releases in the incubator, added new committers and IPMC members, checked off the various requirements to migrate the development process to the ASF, and more generally worked to build this community in the Apache way. Please VOTE to graduate Apache Spark from the incubator. I’ll leave the VOTE open for 96 hours, until February 4th, 2014. The graduation resolution is pasted below. Thanks! Matei and the Apache Spark community snip WHEREAS, the Board of Directors deems it to be in the best interests of the Foundation and consistent with the Foundation's purpose to establish a Project Management Committee charged with the creation and maintenance of open-source software, for distribution at no charge to the public, related to fast and flexible large-scale data analysis on clusters. NOW, THEREFORE, BE IT RESOLVED, that a Project Management Committee (PMC), to be known as the Apache Spark Project, be and hereby is established pursuant to Bylaws of the Foundation; and be it further RESOLVED, that the Apache Spark Project be and hereby is responsible for the creation and maintenance of software related to efficient cluster management, resource isolation and sharing across distributed applications; and be it further RESOLVED, that the office of Vice President, Apache Spark be and hereby is created, the person holding such office to serve at the direction of the Board of Directors as the chair of the Apache Spark Project, and to have primary responsibility for management of the projects within the scope of responsibility of the Apache Spark Project; and be it further RESOLVED, that the persons listed immediately below be and hereby are appointed to serve as the initial members of the Apache Spark Project: * Mosharaf Chowdhury mosha...@apache.org * Jason Dai jason...@apache.org * Tathagata Das t...@eecs.berkeley.edu * Ankur Dave ankurd...@gmail.com * Aaron Davidson aarondavid...@berkeley.edu * Thomas Dudziak to...@apache.org * Robert Evans bo...@apache.org * Thomas Graves tgra...@apache.org * Andy Konwinski and...@apache.org * Stephen Haberman steph...@apache.org * Mark Hamstra markhams...@apache.org * Shane Huang shane_hu...@apache.org * Ryan LeCompte ryanlecom...@apache.org * Haoyuan Li haoy...@apache.org * Sean McNamara mcnam...@apache.org * Mridul Muralidharam mrid...@yahoo-inc.com * Kay Ousterhout k...@eecs.berkeley.edu * Nick Pentreath mln...@apache.org * Imran Rashid im...@quantifind.com * Charles Reiss wog...@apache.org * Josh Rosen joshro...@apache.org * Prashant Sharma prash...@apache.org * Ram Sriharsha harsh...@yahoo-inc.com * Shivaram Venkataraman shiva...@apache.org * Patrick Wendell pwend...@apache.org * Andrew Xia xiajunl...@gmail.com * Reynold Xin r...@apache.org * Matei Zaharia ma...@apache.org NOW, THEREFORE, BE IT FURTHER RESOLVED, that Matei Zaharia be appointed to the office of Vice President, Apache Spark, to serve in accordance with and subject to the direction of the Board of Directors and the Bylaws of the Foundation until death, resignation, retirement, removal or disqualification, or until a successor is appointed; and be it further RESOLVED, that the Apache Spark Project be and hereby is tasked with the migration and rationalization of the Apache Incubator Spark podling; and be it further RESOLVED, that all responsibilities pertaining to the Apache Incubator Spark podling encumbered upon the Apache Incubator Project are hereafter discharged. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc5)
Hey Patrick, We’ve already had two approved Incubator releases and gone through those items, so is it possible to not block the release on this status page? It seems orthogonal to having a release. Matei On Jan 30, 2014, at 3:34 PM, Patrick Hunt ph...@apache.org wrote: -1 due to the status page not being up to date. But otw the sig/xsums looked good, RAT checked out clean and I ran through the release checklist which looks ok. If you fix the status I'll change my vote to +1. A couple nits that I noticed (but not to hold a rel for): * your notice files should be updated for 2014 * the release artifact (spark-0.9.0-incubating.tgz) is missing the DISCLAIMER file, I see that the text is included in the readme, this is somewhat non-standard but seems fine. Patrick On Thu, Jan 30, 2014 at 2:50 PM, Patrick Hunt ph...@apache.org wrote: Hi Patrick, prior to any incubator release you should ensure that your status page is up to date. The Project Setup section in particular: http://incubator.apache.org/projects/spark.html Patrick On Mon, Jan 27, 2014 at 12:02 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark (incubating) version 0.9.0. A vote on this release has passed within the Spark PPMC. The tag to be voted on is v0.9.0-incubating (commit 95d28ff3): https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=95d28ff3d0d20d9c583e184f9e2c5ae842d8a4d9 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The list of keys associated with Spark is available at: https://people.apache.org/keys/group/spark.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1006/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5-docs/ Please vote on releasing this package as Apache Spark 0.9.0-incubating! The vote is open until Thursday, January 30, at 08:05 UTC and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Spark 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.incubator.apache.org/ - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Spark 0.9.0-incubating (rc5)
+1 On Jan 27, 2014, at 12:02 AM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark (incubating) version 0.9.0. A vote on this release has passed within the Spark PPMC. The tag to be voted on is v0.9.0-incubating (commit 95d28ff3): https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=95d28ff3d0d20d9c583e184f9e2c5ae842d8a4d9 The release files, including signatures, digests, etc can be found at: http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5 Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The list of keys associated with Spark is available at: https://people.apache.org/keys/group/spark.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-1006/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-0.9.0-incubating-rc5-docs/ Please vote on releasing this package as Apache Spark 0.9.0-incubating! The vote is open until Thursday, January 30, at 08:05 UTC and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Spark 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.incubator.apache.org/ - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Spark 0.8.0-incubating (rc4)
Hi Marvin, These are not actually files compiled from our source code, but third-party dependencies that are not published to Maven. SBT is a Scala-based build tool where you need that JAR to run. Where exactly is the policy on including these kinds of JARs? The reason we went with them is because we saw other projects do it. For example, Kafka, which also builds using SBT, ships with sbt.jar. Matei On Dec 14, 2013, at 10:58 AM, Marvin Humphrey mar...@rectangular.com wrote: On Sat, Dec 14, 2013 at 10:37 AM, Henry Saputra henry.sapu...@gmail.com wrote: When you said However they both contain binaries, which is not good. were you talking about the spark-0.8.1-incubating-bin-* files ? There seem to be compiled files in the source archive. marvin@knut:~/spark $ tar -zxf spark-0.8.1-incubating.tgz marvin@knut:~/spark $ cd spark-0.8.1-incubating marvin@knut:~/spark/spark-0.8.1-incubating $ find . -print | grep .jar$ ./assembly/lib/net/sf/py4j/py4j/0.7/py4j-0.7.jar ./core/src/test/resources/uncommons-maths-1.2.2.jar ./repl/lib/scala-jline.jar ./sbt/sbt-launch-0.11.3-2.jar ./streaming/lib/org/apache/kafka/kafka/0.7.2-spark/kafka-0.7.2-spark.jar marvin@knut:~/spark/spark-0.8.1-incubating $ One option for addressing that issue would be to move all compiled dependencies to an accompanying -deps archive. Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Spark 0.8.0-incubating (rc4)
Hi Marvin, These are not actually files compiled from our source code, but third-party dependencies that are not published to Maven. SBT is a Scala-based build tool where you need that JAR to run. Where exactly is the policy on including these kinds of JARs? The reason we went with them is because we saw other projects do it. For example, Kafka, which also builds using SBT, ships with sbt.jar. Matei On Dec 14, 2013, at 10:58 AM, Marvin Humphrey mar...@rectangular.com wrote: On Sat, Dec 14, 2013 at 10:37 AM, Henry Saputra henry.sapu...@gmail.com wrote: When you said However they both contain binaries, which is not good. were you talking about the spark-0.8.1-incubating-bin-* files ? There seem to be compiled files in the source archive. marvin@knut:~/spark $ tar -zxf spark-0.8.1-incubating.tgz marvin@knut:~/spark $ cd spark-0.8.1-incubating marvin@knut:~/spark/spark-0.8.1-incubating $ find . -print | grep .jar$ ./assembly/lib/net/sf/py4j/py4j/0.7/py4j-0.7.jar ./core/src/test/resources/uncommons-maths-1.2.2.jar ./repl/lib/scala-jline.jar ./sbt/sbt-launch-0.11.3-2.jar ./streaming/lib/org/apache/kafka/kafka/0.7.2-spark/kafka-0.7.2-spark.jar marvin@knut:~/spark/spark-0.8.1-incubating $ One option for addressing that issue would be to move all compiled dependencies to an accompanying -deps archive. Marvin Humphrey - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Spark 0.8.0-incubating (RC6)
+1 Matei On Sep 20, 2013, at 1:56 PM, Patrick Wendell pwend...@gmail.com wrote: Please vote on releasing the following candidate as Apache Spark (incubating) version 0.8.0. This will be the first incubator release for Spark in Apache. The tag to be voted on is v0.8.0-incubating (commit 3b85a85): https://git-wip-us.apache.org/repos/asf?p=incubator-spark.git;a=commit;h=3b85a8558da2c87873c85f227a189e45bf16b65d The release files, including signatures, digests, etc can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc6/files/ Release artifacts are signed with the following key: https://people.apache.org/keys/committer/pwendell.asc The staging repository for this release can be found at: https://repository.apache.org/content/repositories/orgapachespark-059/ The documentation corresponding to this release can be found at: http://people.apache.org/~pwendell/spark-0.8.0-incubating-rc6/docs/ A vote on this release has passed within the Spark PPMC [1] including +1 votes from our IPMC mentors (Chris Mattman and Henry Saputra). Please vote on releasing this package as Apache Spark 0.8.0-incubating! The vote is open until Monday, September 23rd at 21:00 UTC and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Spark 0.8.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Spark, please see http://spark.incubator.apache.org/ [1] http://mail-archives.apache.org/mod_mbox/incubator-spark-dev/201309.mbox/%3CCABPQxsvS14wfiABj32b_%2BgtLafmDog%3DcbWjn7v4FoqG5g-a7mQ%40mail.gmail.com%3E - Patrick - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [RESULT] [VOTE] Apache Spark for the Incubator
Thanks Chris! We'll get started on all the required steps. Matei On Jun 20, 2013, at 4:35 AM, Mattmann, Chris A (398J) chris.a.mattm...@jpl.nasa.gov wrote: Hi Folks, This VOTE has passed with the following tallies: +1 Chris Mattmann* Konstantin Boudnik Henry Saputra* Reynold Xin Pei Chen Roman Shaposhnik* Suresh Marru* Scott Deboy Ted Dunning* Hitesh Shah Paul Ramirez* Ralph Goers* Alan Cabrera* Thilina Gunarathne Marcel Offermans* Alex Karasulu* Chris Douglas* Andrew Hart* Deepal jayasinghe Ashish Joe Brockmeier* Mohammad Nour El-Din* Arun C Murthy* Tim Williams* Arvind Prabhakar* Matt Franklin* Matei Zaharia Andy Konwinski +0.9 Marvin Humphrey * -indicates IPMC I'll go ahead and get the JIRA tickets filed for email/issue tracking/Git, and then work with the community to get them moving on' over. Thanks for VOTE'ing! Cheers, Chris ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Mattmann, jpluser chris.a.mattm...@jpl.nasa.gov Reply-To: general@incubator.apache.org general@incubator.apache.org Date: Friday, June 7, 2013 10:34 PM To: general@incubator.apache.org general@incubator.apache.org Subject: [VOTE] Apache Spark for the Incubator Hi Folks, OK discussion has died down, time to VOTE to accept Spark into the Apache Incubator. I'll let the VOTE run for at least a week. So far I've heard +1s from the following folks, so no need for them to VOTE again unless they want to change their VOTE: +1 Chris Mattmann* Konstantin Boudnik Henry Saputra* Reynold Xin Pei Chen Roman Shaposhnik* Suresh Marru* * -indicates IPMC [ ] +1 Accept Spark into the Apache Incubator. [ ] +0 Don't care. [ ] -1 Don't accept Spark into the Apache Incubator because.. Proposal text is below. === Abstract === Spark is an open source system for large-scale data analysis on clusters. === Proposal === Spark is an open source system for fast and flexible large-scale data analysis. Spark provides a general purpose runtime that supports low-latency execution in several forms. These include interactive exploration of very large datasets, near real-time stream processing, and ad-hoc SQL analytics (through higher layer extensions). Spark interfaces with HDFS, HBase, Cassandra and several other storage storage layers, and exposes APIs in Scala, Java and Python. Background Spark started as U.C. Berkeley research project, designed to efficiently run machine learning algorithms on large datasets. Over time, it has evolved into a general computing engine as outlined above. Spark¹s developer community has also grown to include additional institutions, such as universities, research labs, and corporations. Funding has been provided by various institutions including the U.S. National Science Foundation, DARPA, and a number of industry sponsors. See: https://amplab.cs.berkeley.edu/sponsors/ for full details. === Rationale === As the number of contributors to Spark has grown, we have sought for a long-term home for the project, and we believe the Apache foundation would be a great fit. Spark is a natural fit for the Apache foundation: Spark already interoperates with several existing Apache projects (HDFS, HBase, Hive, Cassandra, Avro and Flume to name a few). The Spark team is familiar with the Apache process and and subscribes to the Apache mission - the team includes multiple Apache committers already. Finally, joining Apache will help coordinate the development effort of the growing number of organizations which contribute to Spark. == Initial Goals == The initial goals will most likely be to move the existing codebase to Apache and integrate with the Apache development process. Furthermore, we plan for incremental development, and releases along with the Apache guidelines. === Current Status === == Meritocracy == The Spark project already operates on meritocratic principles. Today, Spark has several developers and has accepted multiple major patches from outside of U.C. Berkeley. While this process has remained mostly informal (we do not have an official committer list), an implicit organization exists in which individuals who contribute major components act as maintainers for those modules. If accepted, the Spark project would include several of these participants as committers from the onset. We will work to identify all committers and PPMC members
Re: [VOTE] Apache Spark for the Incubator
/jira/browse/SPARK == Initial Committers == * Matei Zaharia ma...@apache.org * Ankur Dave ankurd...@gmail.com * Tathagata Das t...@eecs.berkeley.edu * Haoyuan Li haoy...@cs.berkeley.edu * Josh Rosen joshro...@cs.berkeley.edu * Reynold Xin r...@cs.berkeley.edu * Shivaram Venkataraman shiva...@eecs.berkeley.edu * Mosharaf Chowdhury mosha...@cs.berkeley.edu * Charles Reiss char...@eecs.berkeley.edu * Andy Konwinski andykonwin...@gmail.com * Patrick Wendell pwend...@eecs.berkeley.edu * Imran Rashid im...@quantifind.com * Ryan LeCompte lecom...@gmail.com * Ravi Pandya ra...@exchange.microsoft.com * Ram Sriharsha harsh...@yahoo-inc.com * Robert Evans ev...@yahoo-inc.com * Mridul Muralidharan mrid...@yahoo-inc.com * Thomas Dudziak to...@clearstorydata.com * Mark Hamstra m...@clearstorydata.com * Stephen Haberman stephen.haber...@gmail.com * Jason Dai jason@intel.com * Shane Huang shannie.hu...@gmail.com * Andrew xia xiajunl...@gmail.com * Nick Pentreath nick.pentre...@gmail.com * Sean McNamara sean.mcnam...@webtrends.com == Affiliations == The initial committers are from nine organizations: UC Berkeley, Quantifind, Microsoft, Yahoo!, ClearStory Data, Bizo, Intel, Mxit and Webtrends. * Matei Zaharia (UCB) * Ankur Dave (UCB) * Tathagata Das (UCB) * Haoyuan Li (UCB) * Josh Rosen (UCB) * Reynold Xin (UCB) * Shivaram Venkataraman (UCB) * Mosharaf Chowdhury (UCB) * Charles Reiss (UCB) * Andy Konwinski (UCB) * Patrick Wendell (UCB) * Imran Rashid (Quantifind) * Ryan LeCompte (Quantifind) * Ravi Pandya (Microsoft) * Ram Sriharsha (Yahoo!) * Robert Evans (Yahoo!) * Mridul Muralidharam (Yahoo!) * Thomas Dudziak (ClearStory) * Mark Hamstra (ClearStory) * Stephen Haberman (Bizo) * Jason Dai (Intel) * Shane Huang (Intel) * Andrew Xia (Intel) * Nick Pentreath (Mxit) * Sean McNamara (Webtrends) == Sponsors == === Champion === * Chris Mattmann === Nominated Mentors === * Chris Mattmann * Paul Ramirez * Andrew Hart * Thomas Dudziak * Suresh Marru * Henry Saputra === Sponsoring Entity === The Apache Incubator ++ Chris Mattmann, Ph.D. Senior Computer Scientist NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 171-266B, Mailstop: 171-246 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Assistant Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.10.0-incubating (RC2)
+1 Tested it on Mac OS X. Matei On Dec 11, 2012, at 12:56 PM, Benjamin Hindman wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.10.0. This will be the second incubator release for Mesos in Apache. The candidate for Mesos 0.10.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.10.0-incubating-RC2/mesos-0.10.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.10.0-incubating-RC2 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.10.0-incubating-RC2/mesos-0.10.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.10.0-incubating-RC2/mesos-0.10.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.10.0-incubating! The vote is open until Friday, December 14th at 5:00 pm (PST) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.10.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://www.mesosproject.org. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
Thanks for taking a look, Matthias! It's very much appreciated. Matei On May 8, 2012, at 2:43 AM, Matthias Wessendorf wrote: Hi, not that I want to (immediately) become a mentor here, but I am happy to oversee future release votes, and see if the community needs some other guidance etc. -M On Fri, May 4, 2012 at 10:32 PM, Jukka Zitting jukka.zitt...@gmail.com wrote: Hi, On Fri, Apr 27, 2012 at 8:47 PM, Benjamin Hindman benjamin.hind...@gmail.com wrote: Can you elaborate on this? Do you mean recruit active participants from the Mesos community into the IPMC? Or do you mean recruit people from the IPMC to be more active in Mesos? Ideally each podling should have at least three active mentors who can make sure that the required threshold of at least three PMC votes for a release is reached. If that's not the case (as it sounds like), there are a few options: * Ask help from other IPMC members to review the particular release candidate. If you're otherwise doing fine, this should be an OK workaround until you graduate. * Find one or more new mentors to replace inactive ones. Based on past experience this can be a bit difficult, but definitely worth a try. * If the above solutions fail, i.e. the Incubator PMC is unable to provide the help and oversight you deserve, we can also promote deserving PPMC members to the IPMC so that they have binding vote on things like releases. This works, but since that's more or less equivalent to saying that at least a part of the PPMC is already able to oversee itself, so one could well argue that a better solution would be to simply let the podling graduate. None of these solutions are really ideal, which is why I'm really hoping to find better ways for us to proactively identify and find solutions to cases where a podling no longer has enough active mentors. Unfortunately that won't help with the pressing matter of your release vote. Any IPMC members around who'd be willing to lend Mesos a hand and review this release candidate? Unless anyone beats me to it (please do! :-), I'll take care of it later in the weekend. BR, Jukka Zitting - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Matthias Wessendorf blog: http://matthiaswessendorf.wordpress.com/ sessions: http://www.slideshare.net/mwessendorf twitter: http://twitter.com/mwessendorf - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
Thanks for taking a look, Tom! Is there anyone else from the IPMC who could take a bit of time to check this over and vote? Matei On Apr 26, 2012, at 5:22 PM, Tom White wrote: +1 checked checksums and signatures; license, notice, disclaimer and svn tag all look correct. Tom On Thu, Apr 19, 2012 at 12:53 PM, Benjamin Hindman b...@berkeley.edu wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the sixth release candidate. Changes since RC4: * Updated NOTICE to include project name and copyright date as well as to include third-party licences. * Changed one of our third-party components to be included as an archive of it's source rather than a binary bundle (Python egg). * Added DISCLAIMER. The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC5 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Monday, April 23rd at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
Thanks for taking a look, Tom! Is there anyone else from the IPMC who could take a bit of time to check this over and vote? Matei On Apr 26, 2012, at 5:22 PM, Tom White wrote: +1 checked checksums and signatures; license, notice, disclaimer and svn tag all look correct. Tom On Thu, Apr 19, 2012 at 12:53 PM, Benjamin Hindman b...@berkeley.edu wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the sixth release candidate. Changes since RC4: * Updated NOTICE to include project name and copyright date as well as to include third-party licences. * Changed one of our third-party components to be included as an archive of it's source rather than a binary bundle (Python egg). * Added DISCLAIMER. The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC5 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Monday, April 23rd at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
Any input on this from the IPMC? We are basically blocking on mentors or IPMC members taking another look, and would really appreciate if someone can do so. Matei On Apr 19, 2012, at 5:21 PM, Matei Zaharia wrote: +1 Tested it on Mac OS X, and it seems to work fine. Matei On Apr 19, 2012, at 12:53 PM, Benjamin Hindman wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the sixth release candidate. Changes since RC4: * Updated NOTICE to include project name and copyright date as well as to include third-party licences. * Changed one of our third-party components to be included as an archive of it's source rather than a binary bundle (Python egg). * Added DISCLAIMER. The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC5 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Monday, April 23rd at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
+1 Tested it on Mac OS X, seems to work fine. Matei On Apr 19, 2012, at 12:53 PM, Benjamin Hindman wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the sixth release candidate. Changes since RC4: * Updated NOTICE to include project name and copyright date as well as to include third-party licences. * Changed one of our third-party components to be included as an archive of it's source rather than a binary bundle (Python egg). * Added DISCLAIMER. The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC5 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Monday, April 23rd at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
Any input on this from the IPMC? We are basically blocking on mentors or IPMC members taking another look, and would really appreciate if someone can do so. Matei On Apr 19, 2012, at 5:21 PM, Matei Zaharia wrote: +1 Tested it on Mac OS X, and it seems to work fine. Matei On Apr 19, 2012, at 12:53 PM, Benjamin Hindman wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the sixth release candidate. Changes since RC4: * Updated NOTICE to include project name and copyright date as well as to include third-party licences. * Changed one of our third-party components to be included as an archive of it's source rather than a binary bundle (Python egg). * Added DISCLAIMER. The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC5 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Monday, April 23rd at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC5)
+1 Tested it on Mac OS X, and it seems to work fine. Matei On Apr 19, 2012, at 12:53 PM, Benjamin Hindman wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the sixth release candidate. Changes since RC4: * Updated NOTICE to include project name and copyright date as well as to include third-party licences. * Changed one of our third-party components to be included as an archive of it's source rather than a binary bundle (Python egg). * Added DISCLAIMER. The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC5 The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC5/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Monday, April 23rd at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC4)
Hi, We'd appreciate some input on this to continue the release process. We can make a set of scripts to download the third-party source tarballs if that's needed. Matei On Apr 10, 2012, at 8:34 AM, Matei Zaharia wrote: Hi Sebb, The third party libraries we included were actually all source code, just compressed in tar.gz files to save space. There were no JARs or binaries. Is distributing their source code okay, or is it still better to get people to download them elsewhere? Matei On Apr 10, 2012, at 6:03 AM, sebb wrote: On 8 April 2012 23:48, Benjamin Hindman b...@berkeley.edu wrote: Hi Sebb, Thanks so much for taking a look! Responses inline. On Sun, Apr 8, 2012 at 2:22 PM, sebb seb...@gmail.com wrote: On 7 April 2012 02:59, Benjamin Hindman b...@berkeley.edu wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the fifth release candidate. Vote thread on mesos-...@incubator.apache.org: http://mail-archives.apache.org/mod_mbox/incubator-mesos-dev/201203.mbox/%3CCAFeOQnW7jk-VcTFpUP-_VonL99JCnggKeNPW%2BGx-Ozdy1U8sjg%40mail.gmail.com%3E The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC4/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC4 The NOTICE file is incomplete; the product name and copyright year(s) are missing. Thanks, I'll add that. The LICENSE file includes references to several 3rd party items. It looks like glog and utilities.cc and others may require mention in NOTICE. Is there a document somewhere which describes when something needs to go into NOTICE in addition to LICENSE? I had read previously read the section titled What Are Required Third-Party Notices? from http://apache.org/legal/resolved.html#required-third-party-notices but I still was unclear whether or not anything needed to be done. AFAIK, it depends on what the license says. There's no DISCLAIMER file in SVN root. Right, I'll add that. There are several 3rd party libraries in SVN under https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC4/third_party/ This was deliberate, so as to make it easier to use for both developers (who check out the repository) and users (who download a distribution). Is there a policy against this? I saw this as analogous to distributing JARs. AIUI, there must be a source-only release. This should have NL files that relate to the source only. A binary release may also be provided, in which case its NL files need to correspond with what is in the binary release. For building from source, the dependencies need to be provided somehow. Various options are possible: - written instructions how to get the dependencies and where to put them - separate script to download the dependencies - build process automatically downloads the dependencies. Depending on the licenses of 3rd party jars you may need to provide a combination of methods. There are some types of dependencies that can only be included if the user makes a deliberate choice of downloading them, see the cited doc. If all the dependencies are binary-distributable, it's probably also OK to provide an archive containing just the dependencies. This must have the relevant NL files. Is the correct next step to cancel this vote, IMO, yes. create a RC5, and start a new vote thread for that? Once the issues have been resolved, yes. Or modify RC4? No, RCs should be immutable. Otherwise how can one tell what one is voting on? Thank you very much! Ben. The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC4/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC4/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Tuesday, April 10th at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] Release Apache Mesos 0.9.0-incubating (RC4)
Hi Sebb, The third party libraries we included were actually all source code, just compressed in tar.gz files to save space. There were no JARs or binaries. Is distributing their source code okay, or is it still better to get people to download them elsewhere? Matei On Apr 10, 2012, at 6:03 AM, sebb wrote: On 8 April 2012 23:48, Benjamin Hindman b...@berkeley.edu wrote: Hi Sebb, Thanks so much for taking a look! Responses inline. On Sun, Apr 8, 2012 at 2:22 PM, sebb seb...@gmail.com wrote: On 7 April 2012 02:59, Benjamin Hindman b...@berkeley.edu wrote: Please vote on releasing the following candidate as Apache Mesos (incubating) version 0.9.0. This will be the first incubator release for Mesos in Apache, but the fifth release candidate. Vote thread on mesos-...@incubator.apache.org: http://mail-archives.apache.org/mod_mbox/incubator-mesos-dev/201203.mbox/%3CCAFeOQnW7jk-VcTFpUP-_VonL99JCnggKeNPW%2BGx-Ozdy1U8sjg%40mail.gmail.com%3E The candidate for Mesos 0.9.0-incubating release is available at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC4/mesos-0.9.0-incubating.tar.gz The tag to be voted on: https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC4 The NOTICE file is incomplete; the product name and copyright year(s) are missing. Thanks, I'll add that. The LICENSE file includes references to several 3rd party items. It looks like glog and utilities.cc and others may require mention in NOTICE. Is there a document somewhere which describes when something needs to go into NOTICE in addition to LICENSE? I had read previously read the section titled What Are Required Third-Party Notices? from http://apache.org/legal/resolved.html#required-third-party-notices but I still was unclear whether or not anything needed to be done. AFAIK, it depends on what the license says. There's no DISCLAIMER file in SVN root. Right, I'll add that. There are several 3rd party libraries in SVN under https://svn.apache.org/repos/asf/incubator/mesos/tags/release-0.9.0-incubating-RC4/third_party/ This was deliberate, so as to make it easier to use for both developers (who check out the repository) and users (who download a distribution). Is there a policy against this? I saw this as analogous to distributing JARs. AIUI, there must be a source-only release. This should have NL files that relate to the source only. A binary release may also be provided, in which case its NL files need to correspond with what is in the binary release. For building from source, the dependencies need to be provided somehow. Various options are possible: - written instructions how to get the dependencies and where to put them - separate script to download the dependencies - build process automatically downloads the dependencies. Depending on the licenses of 3rd party jars you may need to provide a combination of methods. There are some types of dependencies that can only be included if the user makes a deliberate choice of downloading them, see the cited doc. If all the dependencies are binary-distributable, it's probably also OK to provide an archive containing just the dependencies. This must have the relevant NL files. Is the correct next step to cancel this vote, IMO, yes. create a RC5, and start a new vote thread for that? Once the issues have been resolved, yes. Or modify RC4? No, RCs should be immutable. Otherwise how can one tell what one is voting on? Thank you very much! Ben. The MD5 checksum of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC4/mesos-0.9.0-incubating.tar.gz.md5 The signature of the tarball can be found at: http://people.apache.org/~benh/mesos-0.9.0-incubating-RC4/mesos-0.9.0-incubating.tar.gz.asc Mesos' KEYS file, containing the PGP keys used to sign the release: http://svn.apache.org/repos/asf/incubator/mesos/dist/KEYS Please vote on releasing this package as Apache Mesos 0.9.0-incubating! The vote is open until Tuesday, April 10th at 8 pm (a bit more than 72 hours since it's over the weekend) and passes if a majority of at least 3 +1 IPMC votes are cast. [ ] +1 Release this package as Apache Mesos 0.9.0-incubating [ ] -1 Do not release this package because ... To learn more about Apache Mesos, please see http://incubator.apache.org/mesos. - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail:
Re: Podlings needing copyright sign-off
Hi Henri, I believe we got all the required documents to the ASF for the Mesos project. What do we need to do to sign off on this? Matei On Jul 7, 2011, at 10:20 PM, Henri Yandell wrote: Here's a list of the projects in the Incubator who need to sign off their copyright item; namely: Check and make sure that the papers that transfer rights to the ASF been received. It is only necessary to transfer rights for the package, the core code, and any new code produced by the project. The list is: 2007-10-06jspwiki 2008-01-06rat 2008-04-15bluesky (pending retirement) 2008-08-01empire-db 2008-09-23etch 2008-09-29olio 2008-10-01vcl 2008-10-23droids 2008-11-12tashi 2009-02-09kato 2009-02-13stonehenge 2009-05-08ace 2009-05-13socialsite 2009-06-25wink 2009-08-07vxquery 2009-11-08hise 2009-12-15clerezza 2010-01-27manifoldcf 2010-05-19amber 2010-05-21deltacloud 2010-05-24zetacomponents 2010-07-19chukwa 2010-09-05nuvem 2010-09-27alois 2010-11-02celix 2010-11-12kitty 2010-11-24stanbol 2010-12-02jena 2010-12-02opennlp 2010-12-08wave 2011-01-03mesos 2011-02-01easyant 2011-02-05lucene.net 2011-04-30ognl 2011-06-13flume 2011-06-13openofficeorg 2011-06-13sqoop Some are new podlings, so no huge surprise, but others have been around for a long time. I think each podling needs to focus on getting this checklist item resolved. Hen - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [VOTE] [RESULT] Mesos to enter the incubator
Thanks David, this sounds good. Things are a little slow due to the holidays, but we'll get more of the steps done after. Matei On Dec 30, 2010, at 1:29 AM, David Crossley wrote: Matei Zaharia wrote: Thanks everyone for the positive reception! We'll get some help from Tom for the next steps. I did step 1 to add Mesos to the wiki: http://wiki.apache.org/incubator/ReportingSchedule I chose February, March, April. However if you want to report in January too, or to instead report in January, February, March, then go ahead and change it. Now Clutch can help to facilitate your project's way. http://incubator.apache.org/clutch.html -David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
Re: [PROPOSAL] Mesos Project
Thanks everyone for taking a look at the proposal. If there are no questions, I'll probably start a vote thread tomorrow. We're also glad to take on more mentors at this poit if anyone is interested (we have 3 already, but others are welcome). Matei On Dec 16, 2010, at 2:05 AM, Mohammad Nour El-Din nour.moham...@gmail.com wrote: Thanks David for raising this, but IMHO +1 on a proposal is not equal to +1 on a vote, for me it means that the proposal is OK and it is like a probe of proposal acceptance :). On Thu, Dec 16, 2010 at 12:54 AM, Tom White tomwh...@apache.org wrote: Thanks David. I think we can run a separate vote thread once this proposal thread has run its course. Other recent proposal threads seem to have general votes of support on them. As you say, this phase is a time to respond to any questions that are raised, attract more mentors, and so on. Cheers, Tom On Wed, Dec 15, 2010 at 4:12 PM, David Crossley cross...@apache.org wrote: David Crossley wrote: The Proposal phase is for discussing the proposal, encouraging people, attracting mentors, ensuring that the proposal is ready. Then the Vote phase is for voting. Dunno what to do now that it has started :-) They gather steam, and we don't want to re-do it. Perhaps add Vote to the subject line and declare the end time, then tally everything at the end. -David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Thanks - Mohammad Nour Author of (WebSphere Application Server Community Edition 2.0 User Guide) http://www.redbooks.ibm.com/abstracts/sg247585.html - LinkedIn: http://www.linkedin.com/in/mnour - Blog: http://tadabborat.blogspot.com Life is like riding a bicycle. To keep your balance you must keep moving - Albert Einstein Writing clean code is what you must do in order to call yourself a professional. There is no reasonable excuse for doing anything less than your best. - Clean Code: A Handbook of Agile Software Craftsmanship Stay hungry, stay foolish. - Steve Jobs - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[VOTE] Mesos to enter the incubator
in building a solid open source community around Mesos. To work towards an open source community, we have been using the GitHub issue tracker and mailing lists at Berkeley for development discussions within our group for several months now. == Core Developers == Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman, Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from the Swedish Institute of Computer Science (Ali Ghodsi). Although started as a research project, Mesos was always intended to solve operational issues with large clusters and to become an open-source project, building on our successful experience doing research that has been incorporated into Apache Hadoop (several scheduling algorithms). == Alignment == The ASF is a natural host for Mesos given that it is already the home of Hadoop, HBase, Cassandra, and other emerging cloud software projects. Mesos was designed to support Hadoop from the beginning in order to solve operational challenges in Hadoop clusters, and it aims to support a wide range of applications beyond Hadoop as well. Mesos complements the existing Apache cloud computing projects by providing a unified way to manage these systems and to share resources and data between them. = Known Risks = == Orphaned Products == With the current core developers of Mesos being graduate students, there is a risk that these developers will eventually move on to other projects. However, because of the broad scope of Mesos, we all plan to continue working on projects related to it in the next several years. We are also actively working with developers at other organizations, such as Twitter, who are good candidates to become contributors. == Inexperience with Open Source == All of the core developers are active users and followers of open source. Matei Zaharia is a Hadoop committer and has experience with the Apache infrastructure and development process. Andy Konwinski has contributed patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi has released open source software as part of his PhD work that was adopted by a Swedish company. == Homogeneous Developers == The current core developers are all researchers (graduate students and a young professor). However, we hope to establish a developer community that includes contributors from several corporations, and we are already working towards this with Twitter and Facebook. == Reliance on Salaried Developers == Given that the project started in an academic research environment, the core developers are all interested in it primarily for its own sake rather than for the sake of employment. We all intend to continue working on Mesos as volunteers. == Relationships with Other Apache Products == Mesos needs to work well with Hadoop, HBase, and other cloud software projects. Being hosted on the same infrastructure will facilitate this and ultimately help out both Mesos and the projects that can now be managed using it. There is, however, a risk that new projects will be built to run solely on Mesos, introducing a dependency. == An Excessive Fascination with the Apache Brand == While we respect the reputation of the Apache brand and have no doubts that it will attract contributors and users, our interest is primarily to give Mesos a solid home as an open source project following an established development model. Locating the project in Apache will also facilitate collaboration with Hadoop, HBase, and other Apache cluster computing projects, as discussed in the Alignment section. = Documentation = Information about Mesos can be found at http://mesos.berkeley.edu. The following sources may be useful to start with: * Documentation for GitHub release: http://github.com/mesos/mesos/wiki * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011 conference) = Initial Source = Mesos has been under development since spring 2009 by a team of graduate students and researchers. It is currently hosted on GitHub under a BSD license at http://github.com/mesos/mesos. = External Dependencies = The dependencies all have Apache compatible licenses, including BSD, MIT, Boost, and Apache 2.0. = Cryptography = Not applicable. = Required Resources = == Mailing Lists == * mesos-private for private PMC discussions (with moderated subscriptions) * mesos-dev * mesos-commits * mesos-user == Subversion Directory == https://svn.apache.org/repos/asf/incubator/mesos == Issue Tracking == JIRA Mesos (MESOS) == Other Resources == The existing code already has unit tests, so we would like a Hudson instance to run them whenever a new patch is submitted. This can be added after project creation. = Initial Committers = * Ali Ghodsi (ali at sics dot se) * Benjamin Hindman (benh at eecs
Re: [VOTE] Mesos to enter the incubator
, Mesos employs an optimized C++ message-passing library to achieve scalability and supports master failover using Apache ZooKeeper. Mesos already supports running Hadoop and MPI. We plan to add support for other systems as requested (and contributed) by the community. = Current Status = == Meritocracy == Our intent with this incubator proposal is to start building a diverse developer community around Mesos following the Apache meritocracy model. We have wanted to make the project open source and encourage contributors from multiple organizations from the start. We plan to provide plenty of support to new developers and to quickly recruit those who make solid contributions to committer status. == Community == Mesos is currently being used by developers at Twitter and researchers in computer science and civil engineering at Berkeley. We hope to extend the user and developer base further in the future. The current developers and users are all interested in building a solid open source community around Mesos. To work towards an open source community, we have been using the GitHub issue tracker and mailing lists at Berkeley for development discussions within our group for several months now. == Core Developers == Mesos was started by three graduate students at UC Berkeley (Benjamin Hindman, Andy Konwinski and Matei Zaharia), who were soon joined by a postdoc from the Swedish Institute of Computer Science (Ali Ghodsi). Although started as a research project, Mesos was always intended to solve operational issues with large clusters and to become an open-source project, building on our successful experience doing research that has been incorporated into Apache Hadoop (several scheduling algorithms). == Alignment == The ASF is a natural host for Mesos given that it is already the home of Hadoop, HBase, Cassandra, and other emerging cloud software projects. Mesos was designed to support Hadoop from the beginning in order to solve operational challenges in Hadoop clusters, and it aims to support a wide range of applications beyond Hadoop as well. Mesos complements the existing Apache cloud computing projects by providing a unified way to manage these systems and to share resources and data between them. = Known Risks = == Orphaned Products == With the current core developers of Mesos being graduate students, there is a risk that these developers will eventually move on to other projects. However, because of the broad scope of Mesos, we all plan to continue working on projects related to it in the next several years. We are also actively working with developers at other organizations, such as Twitter, who are good candidates to become contributors. == Inexperience with Open Source == All of the core developers are active users and followers of open source. Matei Zaharia is a Hadoop committer and has experience with the Apache infrastructure and development process. Andy Konwinski has contributed patches to Hadoop through the Apache infrastructure as well. Ali Ghodsi has released open source software as part of his PhD work that was adopted by a Swedish company. == Homogeneous Developers == The current core developers are all researchers (graduate students and a young professor). However, we hope to establish a developer community that includes contributors from several corporations, and we are already working towards this with Twitter and Facebook. == Reliance on Salaried Developers == Given that the project started in an academic research environment, the core developers are all interested in it primarily for its own sake rather than for the sake of employment. We all intend to continue working on Mesos as volunteers. == Relationships with Other Apache Products == Mesos needs to work well with Hadoop, HBase, and other cloud software projects. Being hosted on the same infrastructure will facilitate this and ultimately help out both Mesos and the projects that can now be managed using it. There is, however, a risk that new projects will be built to run solely on Mesos, introducing a dependency. == An Excessive Fascination with the Apache Brand == While we respect the reputation of the Apache brand and have no doubts that it will attract contributors and users, our interest is primarily to give Mesos a solid home as an open source project following an established development model. Locating the project in Apache will also facilitate collaboration with Hadoop, HBase, and other Apache cluster computing projects, as discussed in the Alignment section. = Documentation = Information about Mesos can be found at http://mesos.berkeley.edu. The following sources may be useful to start with: * Documentation for GitHub release: http://github.com/mesos/mesos/wiki * Presentation at Hadoop User Group: http://www.cs.berkeley.edu/~matei/talks/2010/hug_mesos.pdf * Tech report on system design and current features: http://mesos.berkeley.edu/mesos_tech_report.pdf (paper to appear at NSDI 2011
Re: [PROPOSAL] Mesos Project
Thanks everyone for taking a look at the proposal. If there are no questions, I'm going to post a vote thread tomorrow. We're also happy to take on more mentors, so let us know if you'd like to be one. Matei On Dec 16, 2010, at 2:05 AM, Mohammad Nour El-Din wrote: Thanks David for raising this, but IMHO +1 on a proposal is not equal to +1 on a vote, for me it means that the proposal is OK and it is like a probe of proposal acceptance :). On Thu, Dec 16, 2010 at 12:54 AM, Tom White tomwh...@apache.org wrote: Thanks David. I think we can run a separate vote thread once this proposal thread has run its course. Other recent proposal threads seem to have general votes of support on them. As you say, this phase is a time to respond to any questions that are raised, attract more mentors, and so on. Cheers, Tom On Wed, Dec 15, 2010 at 4:12 PM, David Crossley cross...@apache.org wrote: David Crossley wrote: The Proposal phase is for discussing the proposal, encouraging people, attracting mentors, ensuring that the proposal is ready. Then the Vote phase is for voting. Dunno what to do now that it has started :-) They gather steam, and we don't want to re-do it. Perhaps add Vote to the subject line and declare the end time, then tally everything at the end. -David - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org -- Thanks - Mohammad Nour Author of (WebSphere Application Server Community Edition 2.0 User Guide) http://www.redbooks.ibm.com/abstracts/sg247585.html - LinkedIn: http://www.linkedin.com/in/mnour - Blog: http://tadabborat.blogspot.com Life is like riding a bicycle. To keep your balance you must keep moving - Albert Einstein Writing clean code is what you must do in order to call yourself a professional. There is no reasonable excuse for doing anything less than your best. - Clean Code: A Handbook of Agile Software Craftsmanship Stay hungry, stay foolish. - Steve Jobs - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org - To unsubscribe, e-mail: general-unsubscr...@incubator.apache.org For additional commands, e-mail: general-h...@incubator.apache.org
[PROPOSAL] Mesos Project
We would like to propose Mesos as an incubator proposal. Mesos is a resource manager for clusters that provides resource sharing and isolation across distributed applications like Apache Hadoop, MPI, or web applications. It started as a research project at UC Berkeley, but is now being used both by other Berkeley groups and at Twitter. We open sourced Mesos in August and would like to grow a broader community around it. Our proposal is included below and available on the wiki at http://wiki.apache.org/incubator/MesosProposal. We look forward to hearing feedback and questions on the proposal. Also, let us know if you're interested in being a mentor. Thanks, Matei Zaharia, Benjamin Hindman, Andy Konwinski, and Ali Ghodsi = Abstract = Mesos is a cluster manager that provides resource sharing and isolation across cluster applications. = Proposal = Mesos is system for sharing resources between cluster applications such as Hadoop MapReduce, HBase, MPI, and web applications. It is motivated by three use cases. First, organizations that use several of these applications can use Mesos to share nodes between them, increasing utilization and simplifying management. Second, inspired by MapReduce, a wide array of new cluster programming frameworks are being proposed, such as Apache Hama, Microsoft Dryad, and Google's Pregel and Caffeine. Mesos provides a common interface for such frameworks to share resources, allowing organizations to use multiple frameworks in the same cluster. Third, Mesos allows users of a framework such as Hadoop to have multiple instances of the framework on the same cluster, facilitating workload isolation and incremental deployment of upgrades. = Background = Mesos was inspired by operational issues experienced in large Apache Hadoop deployments as well as a desire to provide a management system for a wider range of cluster applications. The Apache Hadoop community has long realized that the current model of having one instance of MapReduce control a whole cluster leads to problems with isolation (one job may cause the master to crash, killing all the other jobs), scalability, and software upgrades (an upgrade must be deployed on the whole cluster). Statically partitioning resources into multiple fixed-size MapReduce clusters is unattractive because it lowers both utilization and data locality. The community has discussed a two-level scheduling model where a simple, robust low-level layer enables multiple applications to launch tasks (https://issues.apache.org/jira/browse/MAPREDUCE-279). Mesos is such a layer, with the additional goal of supporting non-Hadoop applications as well. Mesos started as a research project at UC Berkeley, but is now being tested at several companies (including Twitter and Facebook), and has attracted interest from other industry users and researchers as well. We are therefore proposing to place Mesos in the Apache incubator and build an open source community around it. = Rationale = Although a variety of cluster schedulers (e.g. Torque, Sun Grid Engine) already exist in the scientific computing community, they are not well suited for today's data center environment. These schedulers generally give jobs coarse-grained static allocations of the cluster (e.g. X nodes for the full duration of the job). This is problematic because many cluster applications are elastic (can scale up and down), so utilization is not optimal under static partitioning, and because data-intensive applications such as MapReduce need to run a few tasks on every node of the cluster to read data locally. To address these challenges, Mesos is designed around two principles: * Fine-grained sharing: Mesos allocates resources at the level of tasks within a job, allowing applications to scale up and down over time and to take turns accessing data on cluster nodes. * Application-controlled scheduling: Applications control which nodes their tasks run on, allowing them to achieve placement goals such as data locality. In addition to these principles, Mesos is designed to be simple, scalable and robust, becuase a cluster manager must be highly available to support applications and should not become a bottleneck. Application-controlled scheduling already simplifies our design by pushing much of the complex logic of tracking job state to applications. In addition, Mesos employs an optimized C++ message-passing library to achieve scalability and supports master failover using Apache ZooKeeper. Mesos already supports running Hadoop and MPI. We plan to add support for other systems as requested (and contributed) by the community. = Current Status = == Meritocracy == Our intent with this incubator proposal is to start building a diverse developer community around Mesos following the Apache meritocracy model. We have wanted to make the project open source and encourage contributors from multiple organizations from the start. We plan to provide plenty of support
