Re: Foundation policy on releases and Spark nightly builds
This is done, and yes I believe that resolves the issue as far all here know. http://spark.apache.org/downloads.html - https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-NightlyBuilds On Sun, Jul 19, 2015 at 5:26 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, One other thing I'd be okay doing is moving the main text about nightly builds to the wiki and just have header called Nightly builds at the end of the downloads page that says For developers, Spark maintains nightly builds. More information is available on the [Spark developer Wiki](link). I think this would preserve discoverability while also placing the information on the wiki, which seems to be the main ask of the policy. - Patrick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
Thanks, Sean. On Mon, Jul 20, 2015 at 12:22 AM, Sean Owen so...@cloudera.com wrote: This is done, and yes I believe that resolves the issue as far all here know. http://spark.apache.org/downloads.html - https://cwiki.apache.org/confluence/display/SPARK/Useful+Developer+Tools#UsefulDeveloperTools-NightlyBuilds On Sun, Jul 19, 2015 at 5:26 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean, One other thing I'd be okay doing is moving the main text about nightly builds to the wiki and just have header called Nightly builds at the end of the downloads page that says For developers, Spark maintains nightly builds. More information is available on the [Spark developer Wiki](link). I think this would preserve discoverability while also placing the information on the wiki, which seems to be the main ask of the policy. - Patrick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
I am going to make an edit to the download page on the web site to start, as that much seems uncontroversial. Proposed change: Reorder sections to put developer-oriented sections at the bottom, including the info on nightly builds: Download Spark Link with Spark All Releases Spark Source Code Management Nightly Builds Change text to emphasize the audience: Packages are built regularly off of Spark’s master branch and release branches. These provide *Spark developers* access to the bleeding-edge of Spark master or the most recent fixes not yet incorporated into a maintenance release. *They should not be used by anyone except Spark developers, and may be unstable or have serious bugs. End users should only use official releases above. Please subscribe to dev@spark.apache.org if you are a Spark developer to be aware of issues in nightly builds.* Spark nightly packages are available at: On Thu, Jul 16, 2015 at 8:21 AM, Sean Owen so...@cloudera.com wrote: To move this forward, I think one of two things needs to happen: 1. Move this guidance to the wiki. Seems that people gathered here believe that resolves the issue. Done. 2. Put disclaimers on the current downloads page. This may resolve the issue, but then we bring it up on the right mailing list for discussion. It may end up at #1, or may end in a tweak to the policy. I can drive either one. Votes on how to proceed? - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
Sean B., Thank you for giving a thorough reply. I will work with Sean O. and see what we can change to make us more in line with the stated policy. I did some research and it appears that some time between October [1] and December [2] 2006, this page was modified to include stricter policy surrounding nightly builds. Actually, the original version of the policy page encouraged projects to post nightly builds for the benefit of all developers, just as we have been doing. If you detect frustration from the Spark community, it's because this type of situation occurs with some regularity. In this case: (a) A policy exists from ~10 years ago, presumably because some project back then had problematic release management practices and so a policy needed to be created to solve a problem. (b) The policy is outdated now, and no one is 100% sure why it was created (likely many of the people are no longer involved in the ASF who helped craft it). (c) The steps for how to change it are unclear and there isn't clear ownership of the policy document. I think it's unavoidable given the decentralized organization structure of the ASF, but I just want to be up front about our perspective and why you might sense some frustration. [1] https://web.archive.org/web/20061020220358/http://www.apache.org/dev/release.html [2] https://web.archive.org/web/20061231050046/http://www.apache.org/dev/release.html - Patrick On Tue, Jul 14, 2015 at 10:09 AM, Sean Busbey bus...@cloudera.com wrote: Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B, Would you mind outlining for me how we go about changing this policy - I think it's outdated and doesn't make much sense. Ideally I'd like to propose a vote to modify the text slightly such that our current behavior is seen as complaint. Specifically: - Who has the authority to change this document? It's foundation level policy, so I'd presume the board needs to. Since it's part of our legal position, it might be owned by the legal affairs committee[1]. That would mean they could update it without a board resolution. (legal-discuss@ could tell you for sure). - What concrete steps can I take to change the policy? The Legal Affairs Committee is reachable either through their mailing list[2] or their issue tracker[3]. Please be sure to read the entire original document, it explains the rationale that has gone into it. You'll need to address the matters raised there. - You keep mentioning the incubator@ list, why is this the place for such policy to be discussed or decided on? It can't be decided on the general@incubator list, but there are already several relevant parties discussing the matter there. You certainly don't *need* to join that conversation, but the participants there have overlap with the folks who can ultimately decide the issue. Thus, it may help avoid having to repeat things. - What is the reasonable amount of time frame in which the policy change is likely to be decided? I am neither a participant on legal affairs nor the board, so I have no idea. We've had a few times people from the various parts of the ASF come and say we are in violation of a policy. And sometimes other ASF people come and then get in a fight on our mailing list, and there is Please keep in mind that you are also ASF people, as is the entire Spark community (users and all)[4]. Phrasing things in terms of us and them by drawing a distinction on [they] get in a fight on our mailing list is not helpful. back and fourth, and it turns out there isn't so much a widely followed policy as a doc somewhere that is really old and not actually universally followed. It's difficult for us in such situations to now how to proceed and how much autonomy we as a PMC have to make decisions about our own project. Understanding and abiding by ASF legal obligations and policies is the job of each project PMC as a part of their formation by the board[5]. If anyone in your community has questions about what the project can or can not do then it's the job of the PMC find out proactively (rather than take a ask for forgiveness approach). Where the existing documentation is unclear or where you think it might be out of date, you can often get guidance from general@incubator (since it contains a large number of members and folks from across foundation projects) or comdev[6] (since their charter includes explaining ASF policy). If those resources prove insufficient matters can be brought up with either legal-discuss@ or board@. If you find out of date documentation that is not ASF policy, you can have it removed by notifying the appropriate group (i.e. legal-discuss, comdev, or whomever is hosting it). [1]: http://apache.org/legal/ [2]: http://www.apache.org/foundation/mailinglists.html#foundation-legal [3]: https://issues.apache.org/jira/browse/LEGAL/
Re: Foundation policy on releases and Spark nightly builds
Hey Sean, One other thing I'd be okay doing is moving the main text about nightly builds to the wiki and just have header called Nightly builds at the end of the downloads page that says For developers, Spark maintains nightly builds. More information is available on the [Spark developer Wiki](link). I think this would preserve discoverability while also placing the information on the wiki, which seems to be the main ask of the policy. - Patrick On Sun, Jul 19, 2015 at 2:32 AM, Sean Owen so...@cloudera.com wrote: I am going to make an edit to the download page on the web site to start, as that much seems uncontroversial. Proposed change: Reorder sections to put developer-oriented sections at the bottom, including the info on nightly builds: Download Spark Link with Spark All Releases Spark Source Code Management Nightly Builds Change text to emphasize the audience: Packages are built regularly off of Spark’s master branch and release branches. These provide *Spark developers* access to the bleeding-edge of Spark master or the most recent fixes not yet incorporated into a maintenance release. *They should not be used by anyone except Spark developers, and may be unstable or have serious bugs. End users should only use official releases above. Please subscribe to dev@spark.apache.org if you are a Spark developer to be aware of issues in nightly builds.* Spark nightly packages are available at: On Thu, Jul 16, 2015 at 8:21 AM, Sean Owen so...@cloudera.com wrote: To move this forward, I think one of two things needs to happen: 1. Move this guidance to the wiki. Seems that people gathered here believe that resolves the issue. Done. 2. Put disclaimers on the current downloads page. This may resolve the issue, but then we bring it up on the right mailing list for discussion. It may end up at #1, or may end in a tweak to the policy. I can drive either one. Votes on how to proceed? - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
To move this forward, I think one of two things needs to happen: 1. Move this guidance to the wiki. Seems that people gathered here believe that resolves the issue. Done. 2. Put disclaimers on the current downloads page. This may resolve the issue, but then we bring it up on the right mailing list for discussion. It may end up at #1, or may end in a tweak to the policy. I can drive either one. Votes on how to proceed? On Tue, Jul 14, 2015 at 7:39 PM, Sean Busbey bus...@cloudera.com wrote: Point well taken. Allow me to walk back a little and move us in a more productive direction. I can personally empathize with the desire to have nightly builds. I'm a passionate advocate for tight feedback cycles between a project and its downstream users. I am personally involved in several projects that would benefit from nightly builds and would love to see change in this policy, if only to get an example of an implementation that's not buried on a project wiki. My interest here in Spark is two-fold. First, protecting the foundation via its established policies. To this end I periodically look for projects that show up as non-compliant. Second, it seems like the Spark community has some friction with larger ASF processes and I'd like to smooth that out where I can help do so. (I guess this is my way of saying that for better or worse this isn't a drive by ;) ) One reason to throw in with general @ incubator thread is that there are like-minded PPMCs and it is a known location for other PMCs to easily join in. Since you are a PMC you'll have enough standing with legal-discuss to not need someone else on your side of the request, but more PMCs helps to show the demand. We could go directly to legal-discuss with just the question of labeling the nightly section of our download page as developer only. I'm skeptical that this will be accepted given the tone of the guidance document. One possible compromise position is the one taken by Apache Open Office. They have two download pages, one for thr general public and one for project QA and localization volunteers. http://ooo-site.apache.org/download/devbuilds.html I didn't suggest this alternative earlier because I haven't verified yet that it meets the standard of the guidance, and I am reasonably certain that the dev wiki page does. How about we reach out to the OO PMC and see if they've proactively checked? If not we can go as a group to legal-discuss. -- Sean On Jul 14, 2015 12:28 PM, Mark Hamstra m...@clearstorydata.com wrote: Please keep in mind that you are also ASF people, as is the entire Spark community (users and all)[4]. Phrasing things in terms of us and them by drawing a distinction on [they] get in a fight on our mailing list is not helpful. whineBut they started it!/whine A bit more seriously, my perspective is that the Spark community and development process works very well and quite smoothly. The only significant strains that I have witnessed during the time in which Spark has been Apache Spark have been when ASF people who otherwise have neither contributed to Spark development nor participated in the Spark community parachute in to tell us that we are doing things wrong and that we must change our practice in order to conform to their expectations of The Apache Way. Sometimes those admonitions are based on actual Apache bylaws and/or legal requirements, and we need to take them seriously. Other times they have seemed more subjective and have felt more like meddling or stirring up trouble in the community and with a process that is actually working very well. On Tue, Jul 14, 2015 at 10:09 AM, Sean Busbey bus...@cloudera.com wrote: Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B, Would you mind outlining for me how we go about changing this policy - I think it's outdated and doesn't make much sense. Ideally I'd like to propose a vote to modify the text slightly such that our current behavior is seen as complaint. Specifically: - Who has the authority to change this document? It's foundation level policy, so I'd presume the board needs to. Since it's part of our legal position, it might be owned by the legal affairs committee[1]. That would mean they could update it without a board resolution. (legal-discuss@ could tell you for sure). - What concrete steps can I take to change the policy? The Legal Affairs Committee is reachable either through their mailing list[2] or their issue tracker[3]. Please be sure to read the entire original document, it explains the rationale that has gone into it. You'll need to address the matters raised there. - You keep mentioning the incubator@ list, why is this the place for such policy to be discussed or decided on? It can't be decided on the general@incubator list, but there are already several relevant parties discussing
Re: Foundation policy on releases and Spark nightly builds
Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B, Would you mind outlining for me how we go about changing this policy - I think it's outdated and doesn't make much sense. Ideally I'd like to propose a vote to modify the text slightly such that our current behavior is seen as complaint. Specifically: - Who has the authority to change this document? It's foundation level policy, so I'd presume the board needs to. Since it's part of our legal position, it might be owned by the legal affairs committee[1]. That would mean they could update it without a board resolution. (legal-discuss@ could tell you for sure). - What concrete steps can I take to change the policy? The Legal Affairs Committee is reachable either through their mailing list[2] or their issue tracker[3]. Please be sure to read the entire original document, it explains the rationale that has gone into it. You'll need to address the matters raised there. - You keep mentioning the incubator@ list, why is this the place for such policy to be discussed or decided on? It can't be decided on the general@incubator list, but there are already several relevant parties discussing the matter there. You certainly don't *need* to join that conversation, but the participants there have overlap with the folks who can ultimately decide the issue. Thus, it may help avoid having to repeat things. - What is the reasonable amount of time frame in which the policy change is likely to be decided? I am neither a participant on legal affairs nor the board, so I have no idea. We've had a few times people from the various parts of the ASF come and say we are in violation of a policy. And sometimes other ASF people come and then get in a fight on our mailing list, and there is Please keep in mind that you are also ASF people, as is the entire Spark community (users and all)[4]. Phrasing things in terms of us and them by drawing a distinction on [they] get in a fight on our mailing list is not helpful. back and fourth, and it turns out there isn't so much a widely followed policy as a doc somewhere that is really old and not actually universally followed. It's difficult for us in such situations to now how to proceed and how much autonomy we as a PMC have to make decisions about our own project. Understanding and abiding by ASF legal obligations and policies is the job of each project PMC as a part of their formation by the board[5]. If anyone in your community has questions about what the project can or can not do then it's the job of the PMC find out proactively (rather than take a ask for forgiveness approach). Where the existing documentation is unclear or where you think it might be out of date, you can often get guidance from general@incubator (since it contains a large number of members and folks from across foundation projects) or comdev[6] (since their charter includes explaining ASF policy). If those resources prove insufficient matters can be brought up with either legal-discuss@ or board@. If you find out of date documentation that is not ASF policy, you can have it removed by notifying the appropriate group (i.e. legal-discuss, comdev, or whomever is hosting it). [1]: http://apache.org/legal/ [2]: http://www.apache.org/foundation/mailinglists.html#foundation-legal [3]: https://issues.apache.org/jira/browse/LEGAL/ [4]: http://www.apache.org/foundation/how-it-works.html#roles [5]: http://apache.org/foundation/how-it-works.html#pmc [6]: https://community.apache.org/ -- Sean
Re: Foundation policy on releases and Spark nightly builds
Point well taken. Allow me to walk back a little and move us in a more productive direction. I can personally empathize with the desire to have nightly builds. I'm a passionate advocate for tight feedback cycles between a project and its downstream users. I am personally involved in several projects that would benefit from nightly builds and would love to see change in this policy, if only to get an example of an implementation that's not buried on a project wiki. My interest here in Spark is two-fold. First, protecting the foundation via its established policies. To this end I periodically look for projects that show up as non-compliant. Second, it seems like the Spark community has some friction with larger ASF processes and I'd like to smooth that out where I can help do so. (I guess this is my way of saying that for better or worse this isn't a drive by ;) ) One reason to throw in with general @ incubator thread is that there are like-minded PPMCs and it is a known location for other PMCs to easily join in. Since you are a PMC you'll have enough standing with legal-discuss to not need someone else on your side of the request, but more PMCs helps to show the demand. We could go directly to legal-discuss with just the question of labeling the nightly section of our download page as developer only. I'm skeptical that this will be accepted given the tone of the guidance document. One possible compromise position is the one taken by Apache Open Office. They have two download pages, one for thr general public and one for project QA and localization volunteers. http://ooo-site.apache.org/download/devbuilds.html I didn't suggest this alternative earlier because I haven't verified yet that it meets the standard of the guidance, and I am reasonably certain that the dev wiki page does. How about we reach out to the OO PMC and see if they've proactively checked? If not we can go as a group to legal-discuss. -- Sean On Jul 14, 2015 12:28 PM, Mark Hamstra m...@clearstorydata.com wrote: Please keep in mind that you are also ASF people, as is the entire Spark community (users and all)[4]. Phrasing things in terms of us and them by drawing a distinction on [they] get in a fight on our mailing list is not helpful. whineBut they started it!/whine A bit more seriously, my perspective is that the Spark community and development process works very well and quite smoothly. The only significant strains that I have witnessed during the time in which Spark has been Apache Spark have been when ASF people who otherwise have neither contributed to Spark development nor participated in the Spark community parachute in to tell us that we are doing things wrong and that we must change our practice in order to conform to their expectations of The Apache Way. Sometimes those admonitions are based on actual Apache bylaws and/or legal requirements, and we need to take them seriously. Other times they have seemed more subjective and have felt more like meddling or stirring up trouble in the community and with a process that is actually working very well. On Tue, Jul 14, 2015 at 10:09 AM, Sean Busbey bus...@cloudera.com wrote: Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B, Would you mind outlining for me how we go about changing this policy - I think it's outdated and doesn't make much sense. Ideally I'd like to propose a vote to modify the text slightly such that our current behavior is seen as complaint. Specifically: - Who has the authority to change this document? It's foundation level policy, so I'd presume the board needs to. Since it's part of our legal position, it might be owned by the legal affairs committee[1]. That would mean they could update it without a board resolution. (legal-discuss@ could tell you for sure). - What concrete steps can I take to change the policy? The Legal Affairs Committee is reachable either through their mailing list[2] or their issue tracker[3]. Please be sure to read the entire original document, it explains the rationale that has gone into it. You'll need to address the matters raised there. - You keep mentioning the incubator@ list, why is this the place for such policy to be discussed or decided on? It can't be decided on the general@incubator list, but there are already several relevant parties discussing the matter there. You certainly don't *need* to join that conversation, but the participants there have overlap with the folks who can ultimately decide the issue. Thus, it may help avoid having to repeat things. - What is the reasonable amount of time frame in which the policy change is likely to be decided? I am neither a participant on legal affairs nor the board, so I have no idea. We've had a few times people from the various parts of the ASF come and say we are in violation of a policy. And sometimes other
Re: Foundation policy on releases and Spark nightly builds
Please keep in mind that you are also ASF people, as is the entire Spark community (users and all)[4]. Phrasing things in terms of us and them by drawing a distinction on [they] get in a fight on our mailing list is not helpful. whineBut they started it!/whine A bit more seriously, my perspective is that the Spark community and development process works very well and quite smoothly. The only significant strains that I have witnessed during the time in which Spark has been Apache Spark have been when ASF people who otherwise have neither contributed to Spark development nor participated in the Spark community parachute in to tell us that we are doing things wrong and that we must change our practice in order to conform to their expectations of The Apache Way. Sometimes those admonitions are based on actual Apache bylaws and/or legal requirements, and we need to take them seriously. Other times they have seemed more subjective and have felt more like meddling or stirring up trouble in the community and with a process that is actually working very well. On Tue, Jul 14, 2015 at 10:09 AM, Sean Busbey bus...@cloudera.com wrote: Responses inline, with some liberties on ordering. On Sun, Jul 12, 2015 at 10:32 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B, Would you mind outlining for me how we go about changing this policy - I think it's outdated and doesn't make much sense. Ideally I'd like to propose a vote to modify the text slightly such that our current behavior is seen as complaint. Specifically: - Who has the authority to change this document? It's foundation level policy, so I'd presume the board needs to. Since it's part of our legal position, it might be owned by the legal affairs committee[1]. That would mean they could update it without a board resolution. (legal-discuss@ could tell you for sure). - What concrete steps can I take to change the policy? The Legal Affairs Committee is reachable either through their mailing list[2] or their issue tracker[3]. Please be sure to read the entire original document, it explains the rationale that has gone into it. You'll need to address the matters raised there. - You keep mentioning the incubator@ list, why is this the place for such policy to be discussed or decided on? It can't be decided on the general@incubator list, but there are already several relevant parties discussing the matter there. You certainly don't *need* to join that conversation, but the participants there have overlap with the folks who can ultimately decide the issue. Thus, it may help avoid having to repeat things. - What is the reasonable amount of time frame in which the policy change is likely to be decided? I am neither a participant on legal affairs nor the board, so I have no idea. We've had a few times people from the various parts of the ASF come and say we are in violation of a policy. And sometimes other ASF people come and then get in a fight on our mailing list, and there is Please keep in mind that you are also ASF people, as is the entire Spark community (users and all)[4]. Phrasing things in terms of us and them by drawing a distinction on [they] get in a fight on our mailing list is not helpful. back and fourth, and it turns out there isn't so much a widely followed policy as a doc somewhere that is really old and not actually universally followed. It's difficult for us in such situations to now how to proceed and how much autonomy we as a PMC have to make decisions about our own project. Understanding and abiding by ASF legal obligations and policies is the job of each project PMC as a part of their formation by the board[5]. If anyone in your community has questions about what the project can or can not do then it's the job of the PMC find out proactively (rather than take a ask for forgiveness approach). Where the existing documentation is unclear or where you think it might be out of date, you can often get guidance from general@incubator (since it contains a large number of members and folks from across foundation projects) or comdev[6] (since their charter includes explaining ASF policy). If those resources prove insufficient matters can be brought up with either legal-discuss@ or board@. If you find out of date documentation that is not ASF policy, you can have it removed by notifying the appropriate group (i.e. legal-discuss, comdev, or whomever is hosting it). [1]: http://apache.org/legal/ [2]: http://www.apache.org/foundation/mailinglists.html#foundation-legal [3]: https://issues.apache.org/jira/browse/LEGAL/ [4]: http://www.apache.org/foundation/how-it-works.html#roles [5]: http://apache.org/foundation/how-it-works.html#pmc [6]: https://community.apache.org/ -- Sean
Re: Foundation policy on releases and Spark nightly builds
Please note that when the policy refers to developers it means the developers of the project at hand, that is participants on the dev@spark mailing list. As I stated in my original email, you're welcome to continue the discussion on the policy including the definition of developers on general@incubator. But please comply with foundation policy before seeking to have it changed. Just to set expectations, this feature was specifically requested by developers from other projects that integrate with Spark sounds like exactly the kind of thing the policy seeks to prevent. The standing guidance is release more often if downstream projects need to integrate with features faster. On Sun, Jul 12, 2015 at 4:26 PM, Patrick Wendell pwend...@gmail.com wrote: Thanks Sean O. I was thinking something like NOTE: Nightly builds are meant for development and testing purposes. They do not go through Apache's release auditing process and are not official releases. - Patrick On Sun, Jul 12, 2015 at 3:39 PM, Sean Owen so...@cloudera.com wrote: (This sounds pretty good to me. Mark it developers-only, not formally tested by the community, etc.) On Sun, Jul 12, 2015 at 7:50 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B., Thanks for bringing this to our attention. I think putting them on the developer wiki would substantially decrease visibility in a way that is not beneficial to the project - this feature was specifically requested by developers from other projects that integrate with Spark. If the concern underlying that policy is that snapshot builds could be misconstrued as formal releases, I think it would work to put a very clear disclaimer explaining the difference directly adjacent to the link. That's arguably more explicit than just moving the same text to a different page. The formal policy asks us not to include links that encourage non-developers to download the builds. Stating clearly that the audience for those links is developers, in my interpretation that would satisfy the letter and spirit of this policy. - Patrick -- Sean
Re: Foundation policy on releases and Spark nightly builds
(This sounds pretty good to me. Mark it developers-only, not formally tested by the community, etc.) On Sun, Jul 12, 2015 at 7:50 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B., Thanks for bringing this to our attention. I think putting them on the developer wiki would substantially decrease visibility in a way that is not beneficial to the project - this feature was specifically requested by developers from other projects that integrate with Spark. If the concern underlying that policy is that snapshot builds could be misconstrued as formal releases, I think it would work to put a very clear disclaimer explaining the difference directly adjacent to the link. That's arguably more explicit than just moving the same text to a different page. The formal policy asks us not to include links that encourage non-developers to download the builds. Stating clearly that the audience for those links is developers, in my interpretation that would satisfy the letter and spirit of this policy. - Patrick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
Thanks Sean O. I was thinking something like NOTE: Nightly builds are meant for development and testing purposes. They do not go through Apache's release auditing process and are not official releases. - Patrick On Sun, Jul 12, 2015 at 3:39 PM, Sean Owen so...@cloudera.com wrote: (This sounds pretty good to me. Mark it developers-only, not formally tested by the community, etc.) On Sun, Jul 12, 2015 at 7:50 PM, Patrick Wendell pwend...@gmail.com wrote: Hey Sean B., Thanks for bringing this to our attention. I think putting them on the developer wiki would substantially decrease visibility in a way that is not beneficial to the project - this feature was specifically requested by developers from other projects that integrate with Spark. If the concern underlying that policy is that snapshot builds could be misconstrued as formal releases, I think it would work to put a very clear disclaimer explaining the difference directly adjacent to the link. That's arguably more explicit than just moving the same text to a different page. The formal policy asks us not to include links that encourage non-developers to download the builds. Stating clearly that the audience for those links is developers, in my interpretation that would satisfy the letter and spirit of this policy. - Patrick - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
Hey Sean B., Thanks for bringing this to our attention. I think putting them on the developer wiki would substantially decrease visibility in a way that is not beneficial to the project - this feature was specifically requested by developers from other projects that integrate with Spark. If the concern underlying that policy is that snapshot builds could be misconstrued as formal releases, I think it would work to put a very clear disclaimer explaining the difference directly adjacent to the link. That's arguably more explicit than just moving the same text to a different page. The formal policy asks us not to include links that encourage non-developers to download the builds. Stating clearly that the audience for those links is developers, in my interpretation that would satisfy the letter and spirit of this policy. - Patrick On Sat, Jul 11, 2015 at 11:53 AM, Sean Owen so...@cloudera.com wrote: From a developer perspective, I also find it surprising to hear that nightly builds should be hidden from non-developer end users. In an age of Github, what on earth is the problem with distributing the content of master? However I do understand why this exists. To the extent the ASF provides any value, it is at least a legal framework for defining what it means for you and I to give software to a bunch of other people. Software artifacts released according to an ASF process becomes something the ASF can take responsibility for as an entity. Nightly builds are not. It might matter to the committers if, say, somebody commits a serious data loss bug. You don't want to be on the hook individually for putting that into end-user hands. More practically, I think this exists to prevent some projects from lazily depending on unofficial nightly builds as pseudo-releases for long periods of time. End users may come to perceive them as official sanctioned releases when they aren't. That's not the case here of course. I think nightlies aren't for end-users anyway, and I think developers who care would know how to get nightlies anyway. There's little cost to moving this info to the wiki, so I'd do it. On Sat, Jul 11, 2015 at 4:29 PM, Reynold Xin r...@databricks.com wrote: I don't get this rule. It is arbitrary, and does not seem like something that should be enforced at the foundation level. By this reasoning, are we not allowed to list source code management on the project public page as well? The download page clearly states the nightly builds are bleeding-edge. Note that technically we did not violate any rules, since the ones we showed were not nightly builds by the foundation's definition: Nightly Builds are simply built from the Subversion trunk, usually once a day.. Spark nightly artifacts were built from git, not svn trunk. :) (joking). On Sat, Jul 11, 2015 at 7:44 AM, Sean Busbey bus...@cloudera.com wrote: That would be great. A note on that page that it's meant for the use of folks working on the project with a link to your get involved howto would be nice additional context. -- Sean On Jul 11, 2015 6:18 AM, Sean Owen so...@cloudera.com wrote: I suggest we move this info to the developer wiki, to keep it out from the place all and users look for downloads. What do you think about that Sean B? On Sat, Jul 11, 2015 at 5:34 AM, Sean Busbey bus...@cloudera.com wrote: Hi Folks! I noticed that Spark website's download page lists nightly builds and instructions for accessing SNAPSHOT maven artifacts[1]. The ASF policy on releases expressly forbids this kind of publishing outside of the dev@spark community[2]. If you'd like to discuss having the policy updated (including expanding the definition of in the development community), please contribute to the discussion on general@incubator[3] after removing the offending items. [1]: http://spark.apache.org/downloads.html#nightly-packages-and-artifacts [2]: http://www.apache.org/dev/release.html#what [3]: http://s.apache.org/XFP -- Sean - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Foundation policy on releases and Spark nightly builds
From a developer perspective, I also find it surprising to hear that nightly builds should be hidden from non-developer end users. In an age of Github, what on earth is the problem with distributing the content of master? However I do understand why this exists. To the extent the ASF provides any value, it is at least a legal framework for defining what it means for you and I to give software to a bunch of other people. Software artifacts released according to an ASF process becomes something the ASF can take responsibility for as an entity. Nightly builds are not. It might matter to the committers if, say, somebody commits a serious data loss bug. You don't want to be on the hook individually for putting that into end-user hands. More practically, I think this exists to prevent some projects from lazily depending on unofficial nightly builds as pseudo-releases for long periods of time. End users may come to perceive them as official sanctioned releases when they aren't. That's not the case here of course. I think nightlies aren't for end-users anyway, and I think developers who care would know how to get nightlies anyway. There's little cost to moving this info to the wiki, so I'd do it. On Sat, Jul 11, 2015 at 4:29 PM, Reynold Xin r...@databricks.com wrote: I don't get this rule. It is arbitrary, and does not seem like something that should be enforced at the foundation level. By this reasoning, are we not allowed to list source code management on the project public page as well? The download page clearly states the nightly builds are bleeding-edge. Note that technically we did not violate any rules, since the ones we showed were not nightly builds by the foundation's definition: Nightly Builds are simply built from the Subversion trunk, usually once a day.. Spark nightly artifacts were built from git, not svn trunk. :) (joking). On Sat, Jul 11, 2015 at 7:44 AM, Sean Busbey bus...@cloudera.com wrote: That would be great. A note on that page that it's meant for the use of folks working on the project with a link to your get involved howto would be nice additional context. -- Sean On Jul 11, 2015 6:18 AM, Sean Owen so...@cloudera.com wrote: I suggest we move this info to the developer wiki, to keep it out from the place all and users look for downloads. What do you think about that Sean B? On Sat, Jul 11, 2015 at 5:34 AM, Sean Busbey bus...@cloudera.com wrote: Hi Folks! I noticed that Spark website's download page lists nightly builds and instructions for accessing SNAPSHOT maven artifacts[1]. The ASF policy on releases expressly forbids this kind of publishing outside of the dev@spark community[2]. If you'd like to discuss having the policy updated (including expanding the definition of in the development community), please contribute to the discussion on general@incubator[3] after removing the offending items. [1]: http://spark.apache.org/downloads.html#nightly-packages-and-artifacts [2]: http://www.apache.org/dev/release.html#what [3]: http://s.apache.org/XFP -- Sean - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Foundation policy on releases and Spark nightly builds
Hi Folks! I noticed that Spark website's download page lists nightly builds and instructions for accessing SNAPSHOT maven artifacts[1]. The ASF policy on releases expressly forbids this kind of publishing outside of the dev@spark community[2]. If you'd like to discuss having the policy updated (including expanding the definition of in the development community), please contribute to the discussion on general@incubator[3] after removing the offending items. [1]: http://spark.apache.org/downloads.html#nightly-packages-and-artifacts [2]: http://www.apache.org/dev/release.html#what [3]: http://s.apache.org/XFP -- Sean
