On Mon, Jun 8, 2015 at 2:10 AM, Prasad Mujumdar <[email protected]> wrote:
> Hi David, > > Appreciate your initiative to help the community. Thanks! > Please see my response inline. > > thanks > Prasad > > > > On Sat, Jun 6, 2015 at 3:03 PM, David Nalley <[email protected]> wrote: > >> Hi folks, >> >> Warning up front, parsing this mail will take some time, there are >> many external references that will need to be read, feel free to stop >> now and get coffee, tea, or $beverage first. :) >> >> It struck me right before I got on a plane early this morning that we >> were doing a lot of saying there are problems, without providing good >> examples of projects who do it right, so at best it's frustrating, and >> at worse we'll turn it into a Sisyphean task with no end in sight. >> >> So - I specifically want to talk about my concerns about project >> direction. I am going to pull some examples out, if you are affected, >> my apologies, I am not picking on you as an individual, I just want to >> provide explicit case points that we can talk about and compare with >> some other projects: >> >> So - lets starts with SENTRY-621 >> https://issues.apache.org/jira/browse/SENTRY-621 >> https://reviews.apache.org/r/30267/ >> >> So from my perspective - I see the ticket created. It has the very >> simple description: >> "Add new thrift interface for import/export in sentry" ~20 minutes >> later the code was uploaded to review board, and is several thousand >> lines of code, which suggests to me it wasn't developed in the 20 >> intervening minutes. Looking at dev@ mailing list, I can't find a >> discussion about this, which gives me pause to wonder where was it >> decided that this functionality was needed? I also wonder who decided >> that it was needed. >> >> I should also disclaim - I know I have several biases, I have those >> because of projects I've worked on both here at the ASF and elsewhere. >> I also tend to be a bit cynical, and I work for a company that >> develops and sells software, and so my mind interprets the above as: >> $company product manager decided (via whatever process) that a Thrift >> interface for import/export into Sentry was needed. A developer was >> tasked with working on the issue. The project community, appears to >> not have been consulted, but The result is code being developed, >> several thousand lines worth. Then a ticket being created so that it >> could be tracked, and then the patch submitted. Does this make sense? >> I may be completely off base, my biases may be reading into it, and >> I'd be thrilled to be wrong. To reiterate the questions: >> >> 1. Where was it decided that Sentry needed this functionality (link >> please) >> 2. Who decided it? >> 3. When was it decided? >> >> >> So - lets talk about one method of this happening in another Apache >> project: >> >> Implementing Azure in jclouds: >> There are multiple threads on the mailing list about this: >> http://jclouds.markmail.org/thread/2tp3szzv7bqumlzb >> http://jclouds.markmail.org/thread/tr7lmaozq45nvkp2 >> http://jclouds.markmail.org/thread/c37peurxkp6qrazd >> http://jclouds.markmail.org/thread/3kcfdrupmd6he3li >> >> Admittedly this is a much larger feature. It involves many more >> people, but take a look at the differences. >> >> Perhaps a smaller feature is in order, let's look at a smaller one: >> >> This is a bug fix: >> http://cloudstack.markmail.org/thread/n3gn3w4dawivw6gh >> Here's the original user complaining about it on a mailing list: >> http://markmail.org/message/nth7qck27s2chwoc?q=VMsync+issues+with+VRs%3F >> >> Look at the amount of discussion that's going on. Does it bother me >> that SENTRY-621 didn't have that level of discussion. Not inherently, >> but when I see virtually none of that kind of discussion going on for >> weeks at a time, but I see a dev list with virtually nothing but >> Review Requests, it causes me some concern. >> >> And just for the record, jclouds and CloudStack aren't perfect. And >> they occasionally get it wrong. >> Here's a great link if you want to read about one such occurrence of >> it going horribly wrong (and some of the resulting ire): >> >> http://blog.remibergsma.com/2015/05/23/making-xenserver-and-cloudstack-sing-and-dance-together-again/ >> >> The answer to this specific case is that SENTRY-621 is just a subtask of > the the https://issues.apache.org/jira/browse/SENTRY-197. This top level > ticket SENTRY-197 describes the feature fairly well (what, why etc) and > there some community discussion around it on that jira. The contributor > who's working on the feature separated the work into multiple subtask > (which are linked from the top Jira) to make it easier for implementation > and reviews. > BTW, The several thousand lines of code from SENTRY-621 is in fact all > generated java code by Apache Thrift compiler. The real change is just two > lines of interface definition. Since the generated code is part of the > source (to avoid build process dependency on binary artifacts of Apache > Thrift) it's included in the patch. > > In general, it is true that there are far fewer discussions on the mailing > list related to high level project direction. As mentioned in the > incubation proposal [1], Apache Sentry is a access control framework for > Hadoop ecosystem projects. Many of the Sentry contributors are also active > contributors on other Hadoop ecosystem projects. For example, I and Brock > Noland are PMC members of Apache Hive, Greg Channon is PMC member of Apache > Lunic/Solr, Jarcec is PMC member of Apache Sqoop, Lenni Kuff is a active > contributor of Impala and so on. Many of the Sentry features initiated by > such contributors to enable Sentry authorization of these Hadoop projects > which is the core direction the project. This has helped the project to > grow community and the new members have contributed plugins for more Hadoop > projects, for example Apache Sqoop plugin ( > https://issues.apache.org/jira/browse/SENTRY-612). Such ideas and > contributions are discussed lot more frequently on Jira tickets and review > boards which are as open forums as the mailing list. > IMO As the community grows there'll be more ideas on expanding the core > direct and those are likely to generate more mailing list discussion. > > >> Releases: >> >> Releases are another big warning flag, and specifically how the >> project is deciding when to release: >> >> Some projects, have a fixed schedule for releases. Every n weeks they >> release a new version. Some projects wait until there's enough change >> and discuss it. I'm not aware of either happening in Sentry. >> >> Here's a couple of examples: >> This is the 4.4.4 release of CloudStack being discussed: >> http://cloudstack.markmail.org/thread/on3n3mllxoxcc5gb >> >> CloudStack in general strives for a feature release every 4 months (it >> rarely runs on time, but it gives a general sense for people to know >> what to expect). Bug fix releases are generally on demand based on >> feedback from users or developers finding a particularly nasty bug. >> >> >> Jclouds on the other hand is different, feature versions are fluid, >> but planned, while bug fix releases tend to be on a pretty regular >> schedule of a few weeks: >> In example, here's the 1.7.0 timing thread. >> http://jclouds.markmail.org/thread/oju2le7afs6gh2p6 >> >> I can't find anything like that for Sentry. Has Sentry discussed and >> agreed upon a predefined schedule? How does the user community know >> what to expect out of a project for upcoming releases? How would a new >> developer, for instance, know how to plan development work to target a >> specific release? I'd love to find links here, but I am coming up >> empty. The very cynical side of me worries that someone outside of the >> PPMC is deciding 'we need Sentry 1.5.0' or something similar. >> >> That's a fair concern. Let me try to explain a bit more about the factors > that impact the releases. As mentioned earlier , the goal of Apache Sentry > is to be an authorization policy engine for Apache Hadoop and the ecosystem > components like Apache Hive, Apache Pig, Apache HBase, Apache Sqoop etc. > Some these projects have prior authorization framework and Apache Sentry > has implementations of those which can be plugged into such components by > the users. There are other like Apache Sqoop which have recently developed > such framework and Sentry community has developed a plugin for that. This > integration make Apache Sentry dependent on these downstream project > releases. This makes it hard for Sentry to make stable releases fix > schedule with contents that provide any value to the community. For > example, Sentry master branch was dependent Hive 1.2-SNAPSHOT. Once Hive > 1.2 was released [2], the Sentry 1.5 release could proceed. > Correction, it's Apache Hive 1.1 release [4] not 1.2. [4] - http://mail-archives.us.apache.org/mod_mbox/www-announce/201503.mbox/%3CCAASjJOYQHjMxE_57RY3=vf1y_tvvpd0cxybzbpudp3w0f98...@mail.gmail.com%3E > IMO This will be the case at least in short term as more and more Hadoop > project are building their authorization frameworks (eg Hadoop HDFS, Apache > Kafa etc), those communities will likely work with Sentry community to > build authorization plugins in Sentry. > One of the ways to address this is to implement such features on separate > branch so that it won't block the next major release. For example, for > another patch that has a potential dependency on unreleased Hive version is > proposed [3] to be developed on a separate branch. IMO This approach will > enable us to do a more periodic releases. I intend to start that discussion > based on our experience with the work on current trunk. > > >> Thanks for reading this far; I hope this makes sense. If not PLEASE >> ask questions. I am sorry your coffee or tea is now cold :) >> > indeed, but at 2am I am not going for another cup :) > > I'll also say, this is NOT a task list. This is really about >> discussing and defining how the project plans to operate. >> >> Again, appreciate it! Please let us know your comments/suggestions on the > responses. We'll are working towards improving the project. > > [1] - https://wiki.apache.org/incubator/SentryProposal > [2] - > http://mail-archives.us.apache.org/mod_mbox/www-announce/201505.mbox/%[email protected]%3E > [3] - > http://mail-archives.apache.org/mod_mbox/sentry-dev/201506.mbox/%3CCAC-RAHM92GSqDSAnDT2f-FM%3D_09JURdRZMk3xqCiWYWb7mm9tA%40mail.gmail.com%3E > > > --David >> >> On Fri, Jun 5, 2015 at 1:50 PM, Lenni Kuff <[email protected]> wrote: >> > Thanks for putting together this list Prasad, let's start knocking these >> > items off. It seems like there is a lot of low hanging fruit. >> > >> > Thanks, >> > Lenni >> > On Jun 5, 2015 10:44 AM, "Prasad Mujumdar" <[email protected]> >> wrote: >> > >> >> @Mentors, monthly reports are fine if that's helping to show the >> >> progress in the project. >> >> >> >> As discussed on the private@, here's the summary of the action items >> for >> >> the community in order to address the concerns - >> >> - Update the website for status file. - Tracked via SENTRY-759 >> >> - Fix the mailing list archive link on Sentry page - Tracked via >> SENTRY-759 >> >> - Review and Update 'How to xxx' steps - Tracked via SENTRY-759 >> >> - Start discussion thread on dev@ about adding a project roadmap page >> on >> >> wiki - Pending >> >> - Add/improve committer/ppmc responsibilities listing on the wiki - >> Tracked >> >> via SENTRY-760 >> >> - Add release discussion step in how to release - Tracked via >> SENTRY-760 >> >> - Create checklist for how to vote on release - Tracked via SENTRY-760 >> >> >> >> Based on the current feedback, we will add monthly reporting to the >> PPMC >> >> responsibilities. Patrick's suggestion of community members >> volunteering in >> >> advance for next report sounds good. We could try that for next few >> >> reports. >> >> >> >> thanks >> >> Prasad >> >> >> >> >> >> >> >> >> >> On Thu, Jun 4, 2015 at 5:01 PM, Sravya Tirukkovalur < >> [email protected]> >> >> wrote: >> >> >> >> > +1 for the idea. >> >> > >> >> > On Thu, Jun 4, 2015 at 4:33 PM, Patrick Hunt <[email protected]> >> wrote: >> >> > >> >> > > I suspect that not getting the report out early is a side effect >> of not >> >> > > having a specific person assigned to generate the report. Everyone >> >> > assumes >> >> > > everyone else will do it. Perhaps it would help if you identified a >> >> > > particular person, at the start of each period, who is responsible >> for >> >> > the >> >> > > next report? >> >> > > >> >> > > Patrick >> >> > > >> >> > > On Thu, Jun 4, 2015 at 8:38 AM, Sravya Tirukkovalur < >> >> [email protected] >> >> > > >> >> > > wrote: >> >> > > >> >> > > > My sincere apologies for not being able to put together the >> report >> >> much >> >> > > in >> >> > > > advance. I did send out the email one day prior, so that members >> of >> >> all >> >> > > > time zones get a chance to review though, but I agree sooner >> would >> >> have >> >> > > > been definitely better. Also I was not really sure if the >> concerns >> >> > raised >> >> > > > should be listed in the board report or they were mostly for the >> >> > podling >> >> > > > members to keep track, rectify and improve on it. >> >> > > > >> >> > > > Community has started working on the concerns raised and hoping >> to >> >> > > resolve >> >> > > > most of it soon. And I see that most of the product discussions >> >> happen >> >> > on >> >> > > > jira/review board, but I agree that it will be good to start >> >> > discussions >> >> > > > first on dev list for more community involvement. >> >> > > > >> >> > > > As we cannot edit the report now, I think it is fair enough to >> submit >> >> > > > monthly reports for a bit until we resolve all the concerns >> raised. >> >> > > > >> >> > > > Thanks! >> >> > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > On Thu, Jun 4, 2015 at 2:57 AM, Joe Brockmeier <[email protected]> >> >> wrote: >> >> > > > >> >> > > > > On Thu, Jun 4, 2015, at 03:44 AM, David Nalley wrote: >> >> > > > > > Hi folks: >> >> > > > > > >> >> > > > > > I apologize for a lack of engagement of late. That has caused >> >> > > > > > problems. (lack of oversight on the release vote for >> instance) >> >> > > > > > >> >> > > > > > I left comments on the board report about concerns I have, >> and >> >> > wanted >> >> > > > > > to ensure that it's a conversation we continue and that it >> >> doesn't >> >> > > > > > just get lost. >> >> > > > > > >> >> > > > > > So here are my comments - and one of the links will not work >> for >> >> > you, >> >> > > > > > but do work for the board members and IPMC members reviewing >> the >> >> > > board >> >> > > > > > report. My goal is not to beat up, or complain, but ensure >> that >> >> > > things >> >> > > > > > improve and that this projects successfully graduates. >> >> > > > > > >> >> > > > > > dn: Recently during the conversation around graduation, >> mentors >> >> > have >> >> > > > > > raised a number of concerns. The gravest of these is concerns >> >> > > > > > around where project direction is occurring. >> >> > > > > > See http://s.apache.org/sentrygraddiscussion for more >> detail. >> >> > > > > > Additionally, it was discovered that Sentry did not follow >> the >> >> > > > > > Incubator release process for the 1.5.0 release - though >> that is >> >> > now >> >> > > > > > being rectified on general@ - see: >> >> > > > > > http://s.apache.org/sentryreleaseissue >> >> > > > > > I am somewhat concerned that the report mentions neither of >> these >> >> > > > > > things and for that reason, I am withholding signoff. >> >> > > > > >> >> > > > > Full agreement here. Also, I'm disappointed after the feedback >> I've >> >> > > > > given elsewhere that -- once again -- the board report is being >> >> > pulled >> >> > > > > together at the last minute without the opportunity for actual >> >> > > community >> >> > > > > review of the report. The community should have at least 72 >> hours >> >> to >> >> > > > > review something like this for proper discussion. >> >> > > > > >> >> > > > > I see an enormous disconnect between the project's report and >> the >> >> > > > > reality. There are very serious concerns about the podling that >> >> have >> >> > > > > been expressed to the project and they are not reflected here. >> >> > > > > >> >> > > > > > I also think it would be reasonable for the podling to report >> >> > monthly >> >> > > > > > for the next three months or so to report on progress dealing >> >> with >> >> > > > > > these issues. >> >> > > > > >> >> > > > > +1 >> >> > > > > >> >> > > > > > --David >> >> > > > > > >> >> > > > > > >> >> > > > > > >> >> > > > > > On Wed, Jun 3, 2015 at 6:00 PM, Sravya Tirukkovalur < >> >> > > > [email protected] >> >> > > > > > >> >> > > > > > wrote: >> >> > > > > > > Thank you all for proof reading! I just posted the report >> to >> >> > > > incubator >> >> > > > > wiki. >> >> > > > > > > >> >> > > > > > > Regards, >> >> > > > > > > >> >> > > > > > > On Wed, Jun 3, 2015 at 1:56 PM, Gregory Chanan < >> >> > > [email protected] >> >> > > > > >> >> > > > > wrote: >> >> > > > > > > >> >> > > > > > >> lgtm. >> >> > > > > > >> >> >> > > > > > >> Greg >> >> > > > > > >> >> >> > > > > > >> On Wed, Jun 3, 2015 at 12:07 PM, Lenni Kuff < >> >> > [email protected]> >> >> > > > > wrote: >> >> > > > > > >> >> >> > > > > > >> > +1 - lgtm, thanks for writing this up Sravya. >> >> > > > > > >> > >> >> > > > > > >> > >> >> > > > > > >> > On Wed, Jun 3, 2015 at 10:29 AM, Prasad Mujumdar < >> >> > > > > [email protected]> >> >> > > > > > >> > wrote: >> >> > > > > > >> > >> >> > > > > > >> > > Looks fine to me. Thanks Sravya. >> >> > > > > > >> > > +1 >> >> > > > > > >> > > >> >> > > > > > >> > > thanks >> >> > > > > > >> > > Prasad >> >> > > > > > >> > > >> >> > > > > > >> > > >> >> > > > > > >> > > >> >> > > > > > >> > > On Wed, Jun 3, 2015 at 10:20 AM, Sravya Tirukkovalur < >> >> > > > > > >> > [email protected]> >> >> > > > > > >> > > wrote: >> >> > > > > > >> > > >> >> > > > > > >> > > > Thank you Prasad for the feedback! Made the changes. >> >> > > > > > >> > > > >> >> > > > > > >> > > > On Wed, Jun 3, 2015 at 10:06 AM, Prasad Mujumdar < >> >> > > > > > >> [email protected] >> >> > > > > > >> > > >> >> > > > > > >> > > > wrote: >> >> > > > > > >> > > > >> >> > > > > > >> > > > > Thanks Sravya for putting this together! Looks >> >> mostly >> >> > > > > fine. A >> >> > > > > > >> few >> >> > > > > > >> > > > minor >> >> > > > > > >> > > > > comments/suggestions below - >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > bq. Sentry community is getting ready for >> graduation. >> >> > > > > > >> > > > > I am not sure if we want to mention that under >> >> "issues" >> >> > > > > section. >> >> > > > > > >> How >> >> > > > > > >> > > > about >> >> > > > > > >> > > > > adding that to community or project section ? >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > Date of last release - IMO we should wait for IPMC >> >> vote >> >> > > for >> >> > > > > new >> >> > > > > > >> > > release. >> >> > > > > > >> > > > We >> >> > > > > > >> > > > > can mention the new release vote underway in the >> >> > community >> >> > > > > section. >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > thanks >> >> > > > > > >> > > > > Prasad >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > On Tue, Jun 2, 2015 at 4:36 PM, Sravya >> Tirukkovalur < >> >> > > > > > >> > [email protected] >> >> > > > > > >> > > > >> >> > > > > > >> > > > > wrote: >> >> > > > > > >> > > > > >> >> > > > > > >> > > > > > Hi all, >> >> > > > > > >> > > > > > >> >> > > > > > >> > > > > > Here is a draft of report [1]. Please let me >> know if >> >> > you >> >> > > > > have any >> >> > > > > > >> > > > > > suggestions. As we need to submit this today, I >> will >> >> > > wait >> >> > > > > for >> >> > > > > > >> > > another 6 >> >> > > > > > >> > > > > > hours before submitting it on incubator page. >> If I >> >> > > receive >> >> > > > > > >> feedback >> >> > > > > > >> > > > after >> >> > > > > > >> > > > > > that I will update it directly there. >> >> > > > > > >> > > > > > >> >> > > > > > >> > > > > > Thanks! >> >> > > > > > >> > > > > > [1]: >> >> > > > > > >> >> https://cwiki.apache.org/confluence/display/SENTRY/June+2015 >> >> > > > > > >> > > > > > >> >> > > > > > >> > > > > >> >> > > > > > >> > > > >> >> > > > > > >> > > > >> >> > > > > > >> > > > >> >> > > > > > >> > > > -- >> >> > > > > > >> > > > Sravya Tirukkovalur >> >> > > > > > >> > > > >> >> > > > > > >> > > >> >> > > > > > >> > >> >> > > > > > >> >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > >> >> > > > > > > -- >> >> > > > > > > Sravya Tirukkovalur >> >> > > > > >> >> > > > > >> >> > > > > Best, >> >> > > > > >> >> > > > > jzb >> >> > > > > -- >> >> > > > > Joe Brockmeier >> >> > > > > [email protected] >> >> > > > > Twitter: @jzb >> >> > > > > http://www.dissociatedpress.net/ >> >> > > > > >> >> > > > >> >> > > > >> >> > > > >> >> > > > -- >> >> > > > Sravya Tirukkovalur >> >> > > > >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > Sravya Tirukkovalur >> >> > >> >> >> > >
