Re: [VOTE] Release Apache Spark 1.6.3 (RC2)
Heads up that 1.6.3 RC2 might be impacted by the change of the JSON.org licenses to category-x (disallowed dependency license) described in SPARK-18262. Not sure if I'll have time to evaluate in time to cast a non-binding -1 before the voting window closes. - busbey On 2016-11-02 19:40 (-0500), Reynold Xin wrote: > Please vote on releasing the following candidate as Apache Spark version > 1.6.3. The vote is open until Sat, Nov 5, 2016 at 18:00 PDT and passes if a > majority of at least 3+1 PMC votes are cast. > > [ ] +1 Release this package as Apache Spark 1.6.3 > [ ] -1 Do not release this package because ... > > > The tag to be voted on is v1.6.3-rc2 > (1e860747458d74a4ccbd081103a0542a2367b14b) > > This release candidate addresses 52 JIRA tickets: > https://s.apache.org/spark-1.6.3-jira > > The release files, including signatures, digests, etc. can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.6.3-rc2-bin/ > > Release artifacts are signed with the following key: > https://people.apache.org/keys/committer/pwendell.asc > > The staging repository for this release can be found at: > https://repository.apache.org/content/repositories/orgapachespark-1212/ > > The documentation corresponding to this release can be found at: > http://people.apache.org/~pwendell/spark-releases/spark-1.6.3-rc2-docs/ > > > === > == How can I help test this release? > === > If you are a Spark user, you can help us test this release by taking an > existing Spark workload and running on this release candidate, then > reporting any regressions from 1.6.2. > > > == What justifies a -1 vote for this release? > > This is a maintenance release in the 1.6.x series. Bugs already present in > 1.6.2, missing features, or bugs related to new features will not > necessarily block this release. > - To unsubscribe e-mail: dev-unsubscr...@spark.apache.org
Re: [discuss] separate API annotation into two components: InterfaceAudience & InterfaceStability
On Fri, May 13, 2016 at 6:37 AM, Tom Graves wrote: > So we definitely need to be careful here. I know you didn't mention it but > it mentioned by others so I would not recommend using LimitedPrivate. I had > started a discussion on Hadoop about some of this due to the way Spark > needed to use some of the Api's. > https://issues.apache.org/jira/browse/HADOOP-10506 > I think LimitedPrivate gets a bad rap due to the way it is misused in Hadoop. The use case here -- "we offer this to developers of intermediate layers; those willing to update their software as we update ours" -- is a perfectly acceptable distinction from the "this is just for us" and "this is something folks can rely on enough to contract out their software development". Essentially, LimitedPrivate(LIBRARY) or LimitedPrivate(PORCELAIN) (to borrow from git's distinction on interfaces for tool makers vs end users). -- busbey - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: [discuss] separate API annotation into two components: InterfaceAudience & InterfaceStability
We could switch to the Audience Annotation from Apache Yetus[1], and then rely on Public for end-users and LimitedPrivate for those things we intend as lower-level things with particular non-end-user audiences. [1]: http://yetus.apache.org/documentation/in-progress/#yetus-audience-annotations On Thu, May 12, 2016 at 3:35 PM, Reynold Xin wrote: > That's true. I think I want to differentiate end-user vs developer. Public > isn't the best word. Maybe EndUser? > > On Thu, May 12, 2016 at 3:34 PM, Shivaram Venkataraman > wrote: >> >> On Thu, May 12, 2016 at 2:29 PM, Reynold Xin wrote: >> > We currently have three levels of interface annotation: >> > >> > - unannotated: stable public API >> > - DeveloperApi: A lower-level, unstable API intended for developers. >> > - Experimental: An experimental user-facing API. >> > >> > >> > After using this annotation for ~ 2 years, I would like to propose the >> > following changes: >> > >> > 1. Require explicitly annotation for public APIs. This reduces the >> > chance of >> > us accidentally exposing private APIs. >> > >> +1 >> >> > 2. Separate interface annotation into two components: one that describes >> > intended audience, and the other that describes stability, similar to >> > what >> > Hadoop does. This allows us to define "low level" APIs that are stable, >> > e.g. >> > the data source API (I'd argue this is the API that should be more >> > stable >> > than end-user-facing APIs). >> > >> > InterfaceAudience: Public, Developer >> > >> > InterfaceStability: Stable, Experimental >> > >> I'm not very sure about this. What advantage do we get from Public vs. >> Developer ? Also somebody needs to take a judgement call on that which >> might not always be easy to do >> > >> > What do you think? > > -- busbey - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: RFC: Remove "HBaseTest" from examples?
I'd suggest that the hbase-downstreamer project[1] is a better place for folks to see these examples. There's already an example for spark streaming that does not rely on any of the new goodness in the hbase-spark module[2]. Granted, it uses the Spark Java APIs[3], but we'd be glad to have a scala based example if someone wanted to translate. [1]: https://github.com/saintstack/hbase-downstreamer [2]: https://github.com/saintstack/hbase-downstreamer#spark-streaming-test-application [3]: https://s.apache.org/apvQ On Tue, Apr 19, 2016 at 12:59 PM, Ted Yu wrote: > bq. HBase's current support, even if there are bugs or things that still > need to be done, is much better than the Spark example > > In my opinion, a simple example that works is better than a buggy package. > > I hope before long the hbase-spark module in HBase can arrive at a state > which we can advertise as mature - but we're not there yet. > > On Tue, Apr 19, 2016 at 10:50 AM, Marcelo Vanzin > wrote: >> >> You're completely missing my point. I'm saying that HBase's current >> support, even if there are bugs or things that still need to be done, >> is much better than the Spark example, which is basically a call to >> "SparkContext.hadoopRDD". >> >> Spark's example is not helpful in learning how to build an HBase >> application on Spark, and clashes head on with how the HBase >> developers think it should be done. That, and because it brings too >> many dependencies for something that is not really useful, is why I'm >> suggesting removing it. >> >> >> On Tue, Apr 19, 2016 at 10:47 AM, Ted Yu wrote: >> > There is an Open JIRA for fixing the documentation: HBASE-15473 >> > >> > I would say the refguide link you provided should not be considered as >> > complete. >> > >> > Note it is marked as Blocker by Sean B. >> > >> > On Tue, Apr 19, 2016 at 10:43 AM, Marcelo Vanzin >> > wrote: >> >> >> >> You're entitled to your own opinions. >> >> >> >> While you're at it, here's some much better documentation, from the >> >> HBase project themselves, than what the Spark example provides: >> >> http://hbase.apache.org/book.html#spark >> >> >> >> On Tue, Apr 19, 2016 at 10:41 AM, Ted Yu wrote: >> >> > bq. it's actually in use right now in spite of not being in any >> >> > upstream >> >> > HBase release >> >> > >> >> > If it is not in upstream, then it is not relevant for discussion on >> >> > Apache >> >> > mailing list. >> >> > >> >> > On Tue, Apr 19, 2016 at 10:38 AM, Marcelo Vanzin >> >> > >> >> > wrote: >> >> >> >> >> >> Alright, if you prefer, I'll say "it's actually in use right now in >> >> >> spite of not being in any upstream HBase release", and it's more >> >> >> useful than a single example file in the Spark repo for those who >> >> >> really want to integrate with HBase. >> >> >> >> >> >> Spark's example is really very trivial (just uses one of HBase's >> >> >> input >> >> >> formats), which makes it not very useful as a blueprint for >> >> >> developing >> >> >> HBase apps with Spark. >> >> >> >> >> >> On Tue, Apr 19, 2016 at 10:28 AM, Ted Yu >> >> >> wrote: >> >> >> > bq. I wouldn't call it "incomplete". >> >> >> > >> >> >> > I would call it incomplete. >> >> >> > >> >> >> > Please see HBASE-15333 'Enhance the filter to handle short, >> >> >> > integer, >> >> >> > long, >> >> >> > float and double' which is a bug fix. >> >> >> > >> >> >> > Please exclude presence of related of module in vendor distro from >> >> >> > this >> >> >> > discussion. >> >> >> > >> >> >> > Thanks >> >> >> > >> >> >> > On Tue, Apr 19, 2016 at 10:23 AM, Marcelo Vanzin >> >> >> > >> >> >> > wrote: >> >> >> >> >> >> >> >> On Tue, Apr 19, 2016 at 10:20 AM, Ted Yu >> >> >> >> wrote: >> >> >> >> > I want to note that the hbase-spark module in HBase is >> >> >> >> > incomplete. >> >> >> >> > Zhan >> >> >> >> > has >> >> >> >> > several patches pending review. >> >> >> >> >> >> >> >> I wouldn't call it "incomplete". Lots of functionality is there, >> >> >> >> which >> >> >> >> doesn't mean new ones, or more efficient implementations of >> >> >> >> existing >> >> >> >> ones, can't be added. >> >> >> >> >> >> >> >> > hbase-spark module is currently only in master branch which >> >> >> >> > would >> >> >> >> > be >> >> >> >> > released as 2.0 >> >> >> >> >> >> >> >> Just as a side note, it's part of CDH 5.7.0, not that it matters >> >> >> >> much >> >> >> >> for upstream HBase. >> >> >> >> >> >> >> >> -- >> >> >> >> Marcelo >> >> >> > >> >> >> > >> >> >> >> >> >> >> >> >> >> >> >> -- >> >> >> Marcelo >> >> > >> >> > >> >> >> >> >> >> >> >> -- >> >> Marcelo >> > >> > >> >> >> >> -- >> Marcelo > > -- busbey - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: auto closing pull requests that have been inactive > 30 days?
Having a PR closed, especially if due to committers not having hte bandwidth to check on things, will be very discouraging to new folks. Doubly so for those inexperienced with opensource. Even if the message says "feel free to reopen for so-and-so reason", new folks who lack confidence are going to see reopening as "pestering" and busy folks are going to see it as a clear indication that their work is not even valuable enough for a human to give a reason for closing. In either case, the cost of reopening is substantially higher than that button press. How about we start by keeping a report of "at-risk" PRs that have been stale for 30 days to make it easier for committers to look at the prs that have been long inactive? On Mon, Apr 18, 2016 at 2:52 PM, Reynold Xin wrote: > The cost of "reopen" is close to zero, because it is just clicking a button. > I think you were referring to the cost of closing the pull request, and you > are assuming people look at the pull requests that have been inactive for a > long time. That seems equally likely (or unlikely) as committers looking at > the recently closed pull requests. > > In either case, most pull requests are scanned through by us when they are > first open, and if they are important enough, usually they get merged > quickly or a target version is set in JIRA. We can definitely improve that > by making it more explicit. > > > > On Mon, Apr 18, 2016 at 12:46 PM, Ted Yu wrote: >> >> From committers' perspective, would they look at closed PRs ? >> >> If not, the cost is not close to zero. >> Meaning, some potentially useful PRs would never see the light of day. >> >> My two cents. >> >> On Mon, Apr 18, 2016 at 12:43 PM, Reynold Xin wrote: >>> >>> Part of it is how difficult it is to automate this. We can build a >>> perfect engine with a lot of rules that understand everything. But the more >>> complicated rules we need, the more unlikely for any of these to happen. So >>> I'd rather do this and create a nice enough message to tell contributors >>> sometimes mistake happen but the cost to reopen is approximately zero (i.e. >>> click a button on the pull request). >>> >>> >>> On Mon, Apr 18, 2016 at 12:41 PM, Ted Yu wrote: bq. close the ones where they don't respond for a week Does this imply that the script understands response from human ? Meaning, would the script use some regex which signifies that the contributor is willing to close the PR ? If the contributor is willing to close, why wouldn't he / she do it him/herself ? On Mon, Apr 18, 2016 at 12:33 PM, Holden Karau wrote: > > Personally I'd rather err on the side of keeping PRs open, but I > understand wanting to keep the open PRs limited to ones which have a > reasonable chance of being merged. > > What about if we filtered for non-mergeable PRs or instead left a > comment asking the author to respond if they are still available to move > the > PR forward - and close the ones where they don't respond for a week? > > Just a suggestion. > On Monday, April 18, 2016, Ted Yu wrote: >> >> I had one PR which got merged after 3 months. >> >> If the inactivity was due to contributor, I think it can be closed >> after 30 days. >> But if the inactivity was due to lack of review, the PR should be kept >> open. >> >> On Mon, Apr 18, 2016 at 12:17 PM, Cody Koeninger >> wrote: >>> >>> For what it's worth, I have definitely had PRs that sat inactive for >>> more than 30 days due to committers not having time to look at them, >>> but did eventually end up successfully being merged. >>> >>> I guess if this just ends up being a committer ping and reopening the >>> PR, it's fine, but I don't know if it really addresses the underlying >>> issue. >>> >>> On Mon, Apr 18, 2016 at 2:02 PM, Reynold Xin >>> wrote: >>> > We have hit a new high in open pull requests: 469 today. While we >>> > can >>> > certainly get more review bandwidth, many of these are old and >>> > still open >>> > for other reasons. Some are stale because the original authors have >>> > become >>> > busy and inactive, and some others are stale because the committers >>> > are not >>> > sure whether the patch would be useful, but have not rejected the >>> > patch >>> > explicitly. We can cut down the signal to noise ratio by closing >>> > pull >>> > requests that have been inactive for greater than 30 days, with a >>> > nice >>> > message. I just checked and this would close ~ half of the pull >>> > requests. >>> > >>> > For example: >>> > >>> > "Thank you for creating this pull request. Since this pull request >>> > has been >>> > inactive for 30 days, we are automatically closing it. Closing the >>> > pull >>> > request does not remove it from history and will r
Re: Any documentation on Spark's security model beyond YARN?
On Wed, Mar 30, 2016 at 4:33 AM, Steve Loughran wrote: > >> On 29 Mar 2016, at 22:19, Michael Segel wrote: >> >> Hi, >> >> So yeah, I know that Spark jobs running on a Hadoop cluster will inherit its >> security from the underlying YARN job. >> However… that’s not really saying much when you think about some use cases. >> >> Like using the thrift service … >> >> I’m wondering what else is new and what people have been thinking about how >> to enhance spark’s security. >> > > Been thinking a bit. > > One thing to look at is renewal of hbase and hive tokens on long-lived > services, alongside hdfs > > I've been looking at this as well. The current work-around I'm using is to use keytab logins on the executors, which is less than desirable. Since the HBase project maintains Spark integration points, it'd be great if there were just a hook for services to provide "here's how to renew" to a common renewal service. -- busbey - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: IntelliJ license for committers?
the private@spark list is only available to PMC members[1]. Could we document somewhere (the IntelliJ section of the wiki[2]?) both the current point of contact and a list of what happens when things get renewed each year? That way we could include either a note that the POC should email the key to all committers or that committers should reach out to the POC at a given time? [1]: PMC members and ASF Members, so Sean O specifically can search the archives. but that won't solve the general problem. [2]: https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-IntelliJ On Wed, Dec 2, 2015 at 2:09 PM, Josh Rosen wrote: > Yep, I'm the point of contact between us and JetBrains. I forwarded the > 2015 license renewal email to the private@ list, so it should be > accessible via the archives. I'll go ahead and forward you a copy of our > project license, which will have to be renewed in January of next year. > > On Wed, Dec 2, 2015 at 10:24 AM, Sean Owen wrote: > >> Thanks, yes I've seen this, though I recall from another project that >> at some point they said, wait, we already gave your project a license! >> and I had to track down who had it. I think Josh might be the keeper? >> Not a big deal, just making sure I didn't miss an update there. >> >> On Wed, Dec 2, 2015 at 6:18 PM, Yin Huai wrote: >> > I think they can renew your license. In >> > https://www.jetbrains.com/buy/opensource/?product=idea, you can find >> "Update >> > Open Source License". >> > >> > On Wed, Dec 2, 2015 at 7:47 AM, Sean Owen wrote: >> >> >> >> I'm aware that IntelliJ has (at least in the past) made licenses >> >> available to committers in bona fide open source projects, and I >> >> recall they did the same for Spark. I believe I'm using that license >> >> now, but it seems to have expired? If anyone knows the status of that >> >> (or of any renewals to the license), I wonder if you could share that >> >> with me, offline of course. >> >> >> >> - >> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >> > >> > > -- Sean
Re: When to cut RCs
On Wed, Dec 2, 2015 at 3:19 PM, Sean Busbey wrote: > > > On Wed, Dec 2, 2015 at 3:06 PM, Michael Armbrust > wrote: > >> >>> >> The release is theoretically several weeks behind plan on what's >>> intended to be a fixed release cycle too. This is why I'm not sure why >>> today it's suddenly potentially ready for release. >>> >> >> Up until today various committers have told me that there were known >> issues with branch-1.6 that would cause them to -1 the release. Whenever >> this happened, I asked them to ensure there was a properly targeted blocker >> JIRA open so people could publicly track the status of the release. As >> long as such issues were open, I only published a preview since making an >> RC is pretty high cost. >> >> I'm sorry that it felt sudden to you, but as of last night all such known >> issues were resolved and thus I cut a release as soon as this was the case. >> >> > > It would help me, and I'm sure other less-active folks, if this kind of > feedback cycle were visible on dev@spark. It would also have the benefit > of getting feedback from non-committers who might e.g. have some issue that > they see in their own production deployments. > > > Sorry for a second email so soon. I meant to also ask, what keeps the cost of making an RC high? Can we bring it down with better tooling? -- Sean
Re: When to cut RCs
On Wed, Dec 2, 2015 at 3:06 PM, Michael Armbrust wrote: > >> > The release is theoretically several weeks behind plan on what's >> intended to be a fixed release cycle too. This is why I'm not sure why >> today it's suddenly potentially ready for release. >> > > Up until today various committers have told me that there were known > issues with branch-1.6 that would cause them to -1 the release. Whenever > this happened, I asked them to ensure there was a properly targeted blocker > JIRA open so people could publicly track the status of the release. As > long as such issues were open, I only published a preview since making an > RC is pretty high cost. > > I'm sorry that it felt sudden to you, but as of last night all such known > issues were resolved and thus I cut a release as soon as this was the case. > > It would help me, and I'm sure other less-active folks, if this kind of feedback cycle were visible on dev@spark. It would also have the benefit of getting feedback from non-committers who might e.g. have some issue that they see in their own production deployments. -- Sean
Re: Foundation policy on releases and Spark nightly builds
Looks good to me. Thanks for helping find a common ground everyone, and Sean for handling the implementation. On Mon, Jul 20, 2015 at 2:22 AM, Sean Owen wrote: > This is done, and yes I believe that resolves the issue as far all here > know. > > http://spark.apache.org/downloads.html > -> > > https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-NightlyBuilds > > On Sun, Jul 19, 2015 at 5:26 PM, Patrick Wendell > wrote: > > Hey Sean, > > > > One other thing I'd be okay doing is moving the main text about > > nightly builds to the wiki and just have header called "Nightly > > builds" at the end of the downloads page that says "For developers, > > Spark maintains nightly builds. More information is available on the > > [Spark developer Wiki](link)". I think this would preserve > > discoverability while also placing the information on the wiki, which > > seems to be the main ask of the policy. > > > > - Patrick > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > > -- Sean
Re: Foundation policy on releases and Spark nightly builds
Point well taken. Allow me to walk back a little and move us in a more productive direction. I can personally empathize with the desire to have nightly builds. I'm a passionate advocate for tight feedback cycles between a project and its downstream users. I am personally involved in several projects that would benefit from nightly builds and would love to see change in this policy, if only to get an example of an implementation that's not buried on a project wiki. My interest here in Spark is two-fold. First, protecting the foundation via its established policies. To this end I periodically look for projects that show up as non-compliant. Second, it seems like the Spark community has some friction with larger ASF processes and I'd like to smooth that out where I can help do so. (I guess this is my way of saying that for better or worse this isn't a drive by ;) ) One reason to throw in with general @ incubator thread is that there are like-minded PPMCs and it is a known location for other PMCs to easily join in. Since you are a PMC you'll have enough standing with legal-discuss to not need someone else on your side of the request, but more PMCs helps to show the demand. We could go directly to legal-discuss with just the question of labeling the nightly section of our download page as "developer only." I'm skeptical that this will be accepted given the tone of the guidance document. One possible compromise position is the one taken by Apache Open Office. They have two download pages, one for thr general public and one for project QA and localization volunteers. http://ooo-site.apache.org/download/devbuilds.html I didn't suggest this alternative earlier because I haven't verified yet that it meets the standard of the guidance, and I am reasonably certain that the dev wiki page does. How about we reach out to the OO PMC and see if they've proactively checked? If not we can go as a group to legal-discuss. -- Sean On Jul 14, 2015 12:28 PM, "Mark Hamstra" wrote: > Please keep in mind that you are also "ASF people," as is the entire Spark >> community (users and all)[4]. Phrasing things in terms of "us and them" by >> drawing a distinction on "[they] get in a fight on our mailing list" is not >> helpful. > > But they started it! > > A bit more seriously, my perspective is that the Spark community and > development process works very well and quite smoothly. The only > significant strains that I have witnessed during the time in which Spark > has been Apache Spark have been when "ASF people" who otherwise have > neither contributed to Spark development nor participated in the Spark > community parachute in to tell us that we are doing things wrong and that > we must change our practice in order to conform to their expectations of > "The Apache Way". Sometimes those admonitions are based on actual Apache > bylaws and/or legal requirements, and we need to take them seriously. > Other times they have seemed more subjective and have felt more like > meddling or stirring up trouble in the community and with a process that is > actually working very well. > > On Tue, Jul 14, 2015 at 10:09 AM, Sean Busbey wrote: > >> Responses inline, with some liberties on ordering. >> >> On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell >> wrote: >> >>> Hey Sean B, >>> >>> Would you mind outlining for me how we go about changing this policy - >>> I think it's outdated and doesn't make much sense. Ideally I'd like to >>> propose a vote to modify the text slightly such that our current >>> behavior is seen as complaint. Specifically: >>> >> >> >> >>> - Who has the authority to change this document? >>> >> >> It's foundation level policy, so I'd presume the board needs to. Since >> it's part of our legal position, it might be owned by the legal affairs >> committee[1]. That would mean they could update it without a board >> resolution. (legal-discuss@ could tell you for sure). >> >> >>> - What concrete steps can I take to change the policy? >>> >> >> The Legal Affairs Committee is reachable either through their mailing >> list[2] or their issue tracker[3]. >> >> Please be sure to read the entire original document, it explains the >> rationale that has gone into it. You'll need to address the matters raised >> there. >> >> >> >>> - You keep mentioning the incubator@ list, why is this the place for >>> such policy to be discussed or decided on? >>> >> >> >> It can't be decided on the general@incubator lis
Re: Foundation policy on releases and Spark nightly builds
Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell wrote: > Hey Sean B, > > Would you mind outlining for me how we go about changing this policy - > I think it's outdated and doesn't make much sense. Ideally I'd like to > propose a vote to modify the text slightly such that our current > behavior is seen as complaint. Specifically: > > - Who has the authority to change this document? > It's foundation level policy, so I'd presume the board needs to. Since it's part of our legal position, it might be owned by the legal affairs committee[1]. That would mean they could update it without a board resolution. (legal-discuss@ could tell you for sure). > - What concrete steps can I take to change the policy? > The Legal Affairs Committee is reachable either through their mailing list[2] or their issue tracker[3]. Please be sure to read the entire original document, it explains the rationale that has gone into it. You'll need to address the matters raised there. > - You keep mentioning the incubator@ list, why is this the place for > such policy to be discussed or decided on? > It can't be decided on the general@incubator list, but there are already several relevant parties discussing the matter there. You certainly don't *need* to join that conversation, but the participants there have overlap with the folks who can ultimately decide the issue. Thus, it may help avoid having to repeat things. > - What is the reasonable amount of time frame in which the policy > change is likely to be decided? > > I am neither a participant on legal affairs nor the board, so I have no idea. > We've had a few times people from the various parts of the ASF come > and say we are in violation of a policy. And sometimes other ASF > people come and then get in a fight on our mailing list, and there is > Please keep in mind that you are also "ASF people," as is the entire Spark community (users and all)[4]. Phrasing things in terms of "us and them" by drawing a distinction on "[they] get in a fight on our mailing list" is not helpful. > back and fourth, and it turns out there isn't so much a widely > followed policy as a doc somewhere that is really old and not actually > universally followed. It's difficult for us in such situations to now > how to proceed and how much autonomy we as a PMC have to make > decisions about our own project. > > Understanding and abiding by ASF legal obligations and policies is the job of each project PMC as a part of their formation by the board[5]. If anyone in your community has questions about what the project can or can not do then it's the job of the PMC find out proactively (rather than take a "ask for forgiveness" approach). Where the existing documentation is unclear or where you think it might be out of date, you can often get guidance from general@incubator (since it contains a large number of members and folks from across foundation projects) or comdev[6] (since their charter includes explaining ASF policy). If those resources prove insufficient matters can be brought up with either legal-discuss@ or board@. If you find out of date documentation that is not ASF policy, you can have it removed by notifying the appropriate group (i.e. legal-discuss, comdev, or whomever is hosting it). [1]: http://apache.org/legal/ [2]: http://www.apache.org/foundation/mailinglists.html#foundation-legal [3]: https://issues.apache.org/jira/browse/LEGAL/ [4]: http://www.apache.org/foundation/how-it-works.html#roles [5]: http://apache.org/foundation/how-it-works.html#pmc [6]: https://community.apache.org/ -- Sean
Re: Foundation policy on releases and Spark nightly builds
Please note that when the policy refers to "developers" it means the developers of the project at hand, that is participants on the dev@spark mailing list. As I stated in my original email, you're welcome to continue the discussion on the policy including the definition of developers on general@incubator. But please comply with foundation policy before seeking to have it changed. Just to set expectations, "this feature was specifically requested by developers from other projects that integrate with Spark" sounds like exactly the kind of thing the policy seeks to prevent. The standing guidance is "release more often" if downstream projects need to integrate with features faster. On Sun, Jul 12, 2015 at 4:26 PM, Patrick Wendell wrote: > Thanks Sean O. I was thinking something like "NOTE: Nightly builds are > meant for development and testing purposes. They do not go through > Apache's release auditing process and are not official releases." > > - Patrick > > On Sun, Jul 12, 2015 at 3:39 PM, Sean Owen wrote: > > (This sounds pretty good to me. Mark it developers-only, not formally > > tested by the community, etc.) > > > > On Sun, Jul 12, 2015 at 7:50 PM, Patrick Wendell > wrote: > >> Hey Sean B., > >> > >> Thanks for bringing this to our attention. I think putting them on the > >> developer wiki would substantially decrease visibility in a way that > >> is not beneficial to the project - this feature was specifically > >> requested by developers from other projects that integrate with Spark. > >> > >> If the concern underlying that policy is that snapshot builds could be > >> misconstrued as formal releases, I think it would work to put a very > >> clear disclaimer explaining the difference directly adjacent to the > >> link. That's arguably more explicit than just moving the same text to > >> a different page. > >> > >> The formal policy asks us not to include links "that encourage > >> non-developers to download" the builds. Stating clearly that the > >> audience for those links is developers, in my interpretation that > >> would satisfy the letter and spirit of this policy. > >> > >> - Patrick > >> > -- Sean
Re: Foundation policy on releases and Spark nightly builds
That would be great. A note on that page that it's meant for the use of folks working on the project with a link to your "get involved" howto would be nice additional context. -- Sean On Jul 11, 2015 6:18 AM, "Sean Owen" wrote: > I suggest we move this info to the developer wiki, to keep it out from > the place all and users look for downloads. What do you think about > that Sean B? > > On Sat, Jul 11, 2015 at 5:34 AM, Sean Busbey wrote: > > Hi Folks! > > > > I noticed that Spark website's download page lists nightly builds and > > instructions for accessing SNAPSHOT maven artifacts[1]. The ASF policy on > > releases expressly forbids this kind of publishing outside of the > dev@spark > > community[2]. > > > > If you'd like to discuss having the policy updated (including expanding > the > > definition of "in the development community"), please contribute to the > > discussion on general@incubator[3] after removing the offending items. > > > > [1]: > http://spark.apache.org/downloads.html#nightly-packages-and-artifacts > > [2]: http://www.apache.org/dev/release.html#what > > [3]: http://s.apache.org/XFP > > > > -- > > Sean >
Foundation policy on releases and Spark nightly builds
Hi Folks! I noticed that Spark website's download page lists nightly builds and instructions for accessing SNAPSHOT maven artifacts[1]. The ASF policy on releases expressly forbids this kind of publishing outside of the dev@spark community[2]. If you'd like to discuss having the policy updated (including expanding the definition of "in the development community"), please contribute to the discussion on general@incubator[3] after removing the offending items. [1]: http://spark.apache.org/downloads.html#nightly-packages-and-artifacts [2]: http://www.apache.org/dev/release.html#what [3]: http://s.apache.org/XFP -- Sean
