Re: Keep or remove Debian packaging in Spark?
Yeah, I'm fine with that. On Mon, Feb 9, 2015 at 10:09 PM, Patrick Wendell wrote: > Mark was involved in adding this code (IIRC) and has also been the > most active in maintaining it. So I'd be interested in hearing his > thoughts on that proposal. Mark - would you be okay deprecating this > and having Spark instead work with the upstream projects that focus on > packaging? > > My feeling is that it's better to just have nothing than to have > something not usable out-of-the-box (which to your point, is a lot > more work). > > On Mon, Feb 9, 2015 at 4:10 PM, wrote: > > This could be something if the spark community wanted to not maintain > debs/rpms directly via the project could direct interested efforts towards > apache bigtop. Right now debs/rpms of bigtop components, as well as > related tests is a focus. > > > > Something that would be great is if at least one spark committer with > interests in config/pkg/testing could be liason and pt for bigtop efforts. > > > > Right now focus on bigtop 0.9, which currently includes spark 1.2. Jira > for items included in 0.9 can be found here: > > > > https://issues.apache.org/jira/browse/BIGTOP-1480 > > > > > > > > -Original Message- > > From: Sean Owen [mailto:so...@cloudera.com] > > Sent: Monday, February 9, 2015 3:52 PM > > To: Nicholas Chammas > > Cc: Patrick Wendell; Mark Hamstra; dev > > Subject: Re: Keep or remove Debian packaging in Spark? > > > > What about this straw man proposal: deprecate in 1.3 with some kind of > message in the build, and remove for 1.4? And add a pointer to any > third-party packaging that might provide similar functionality? > > > > On Mon, Feb 9, 2015 at 6:47 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> +1 to an "official" deprecation + redirecting users to some other > >> +project > >> that will or already is taking this on. > >> > >> Nate? > >> > >> > >> > >> On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell > >> wrote: > >>> > >>> I have wondered whether we should sort of deprecated it more > >>> officially, since otherwise I think people have the reasonable > >>> expectation based on the current code that Spark intends to support > >>> "complete" Debian packaging as part of the upstream build. Having > >>> something that's sort-of maintained but no one is helping review and > >>> merge patches on it or make it fully functional, IMO that doesn't > >>> benefit us or our users. There are a bunch of other projects that are > >>> specifically devoted to packaging, so it seems like there is a clear > >>> separation of concerns here. > >>> > >>> On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra > >>> > >>> wrote: > >>> >> > >>> >> it sounds like nobody intends these to be used to actually deploy > >>> >> Spark > >>> > > >>> > > >>> > I wouldn't go quite that far. What we have now can serve as useful > >>> > input to a deployment tool like Chef, but the user is then going to > >>> > need to add some customization or configuration within the context > >>> > of that tooling to get Spark installed just the way they want. So > >>> > it is not so much that the current Debian packaging can't be used > >>> > as that it has never really been intended to be a completely > >>> > finished product that a newcomer could, for example, use to install > >>> > Spark completely and quickly to Ubuntu and have a fully-functional > >>> > environment in which they could then run all of the examples, > >>> > tutorials, etc. > >>> > > >>> > Getting to that level of packaging (and maintenance) is something > >>> > that I'm not sure we want to do since that is a better fit with > >>> > Bigtop and the efforts of Cloudera, Horton Works, MapR, etc. to > >>> > distribute Spark. > >>> > > >>> > On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen > wrote: > >>> > > >>> >> This is a straw poll to assess whether there is support to keep > >>> >> and fix, or remove, the Debian packaging-related config in Spark. > >>> >> > >>> >> I see several oldish outstanding JIRAs relating to problems in the > >>> >> pack
Re: Keep or remove Debian packaging in Spark?
@patrick @nate good idea, might as well join forces... right now in bigtop we already have - packaging of both deb and rpm versions of spark in bigtop, + - puppet recipes which work for standalone deployment, + - curation of e2e vagrant tests + bigpetstore-spark, for automated testing spark in both docker and VMs. - builds for YARN, hive, and so on. We have a maintainers.txt file which we would be happy to add folks from spark contrib to, if there is interest in coordinating packaging. On Tue, Feb 10, 2015 at 1:09 AM, Patrick Wendell wrote: > Mark was involved in adding this code (IIRC) and has also been the > most active in maintaining it. So I'd be interested in hearing his > thoughts on that proposal. Mark - would you be okay deprecating this > and having Spark instead work with the upstream projects that focus on > packaging? > > My feeling is that it's better to just have nothing than to have > something not usable out-of-the-box (which to your point, is a lot > more work). > > On Mon, Feb 9, 2015 at 4:10 PM, wrote: > > This could be something if the spark community wanted to not maintain > debs/rpms directly via the project could direct interested efforts towards > apache bigtop. Right now debs/rpms of bigtop components, as well as > related tests is a focus. > > > > Something that would be great is if at least one spark committer with > interests in config/pkg/testing could be liason and pt for bigtop efforts. > > > > Right now focus on bigtop 0.9, which currently includes spark 1.2. Jira > for items included in 0.9 can be found here: > > > > https://issues.apache.org/jira/browse/BIGTOP-1480 > > > > > > > > -Original Message- > > From: Sean Owen [mailto:so...@cloudera.com] > > Sent: Monday, February 9, 2015 3:52 PM > > To: Nicholas Chammas > > Cc: Patrick Wendell; Mark Hamstra; dev > > Subject: Re: Keep or remove Debian packaging in Spark? > > > > What about this straw man proposal: deprecate in 1.3 with some kind of > message in the build, and remove for 1.4? And add a pointer to any > third-party packaging that might provide similar functionality? > > > > On Mon, Feb 9, 2015 at 6:47 PM, Nicholas Chammas < > nicholas.cham...@gmail.com> wrote: > >> +1 to an "official" deprecation + redirecting users to some other > >> +project > >> that will or already is taking this on. > >> > >> Nate? > >> > >> > >> > >> On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell > >> wrote: > >>> > >>> I have wondered whether we should sort of deprecated it more > >>> officially, since otherwise I think people have the reasonable > >>> expectation based on the current code that Spark intends to support > >>> "complete" Debian packaging as part of the upstream build. Having > >>> something that's sort-of maintained but no one is helping review and > >>> merge patches on it or make it fully functional, IMO that doesn't > >>> benefit us or our users. There are a bunch of other projects that are > >>> specifically devoted to packaging, so it seems like there is a clear > >>> separation of concerns here. > >>> > >>> On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra > >>> > >>> wrote: > >>> >> > >>> >> it sounds like nobody intends these to be used to actually deploy > >>> >> Spark > >>> > > >>> > > >>> > I wouldn't go quite that far. What we have now can serve as useful > >>> > input to a deployment tool like Chef, but the user is then going to > >>> > need to add some customization or configuration within the context > >>> > of that tooling to get Spark installed just the way they want. So > >>> > it is not so much that the current Debian packaging can't be used > >>> > as that it has never really been intended to be a completely > >>> > finished product that a newcomer could, for example, use to install > >>> > Spark completely and quickly to Ubuntu and have a fully-functional > >>> > environment in which they could then run all of the examples, > >>> > tutorials, etc. > >>> > > >>> > Getting to that level of packaging (and maintenance) is something > >>> > that I'm not sure we want to do since that is a better fit with > >>> > Bigtop and the efforts of Cloudera, Horton Works, MapR, etc. to > >>> > distribute Spa
Re: Keep or remove Debian packaging in Spark?
Mark was involved in adding this code (IIRC) and has also been the most active in maintaining it. So I'd be interested in hearing his thoughts on that proposal. Mark - would you be okay deprecating this and having Spark instead work with the upstream projects that focus on packaging? My feeling is that it's better to just have nothing than to have something not usable out-of-the-box (which to your point, is a lot more work). On Mon, Feb 9, 2015 at 4:10 PM, wrote: > This could be something if the spark community wanted to not maintain > debs/rpms directly via the project could direct interested efforts towards > apache bigtop. Right now debs/rpms of bigtop components, as well as related > tests is a focus. > > Something that would be great is if at least one spark committer with > interests in config/pkg/testing could be liason and pt for bigtop efforts. > > Right now focus on bigtop 0.9, which currently includes spark 1.2. Jira for > items included in 0.9 can be found here: > > https://issues.apache.org/jira/browse/BIGTOP-1480 > > > > -Original Message- > From: Sean Owen [mailto:so...@cloudera.com] > Sent: Monday, February 9, 2015 3:52 PM > To: Nicholas Chammas > Cc: Patrick Wendell; Mark Hamstra; dev > Subject: Re: Keep or remove Debian packaging in Spark? > > What about this straw man proposal: deprecate in 1.3 with some kind of > message in the build, and remove for 1.4? And add a pointer to any > third-party packaging that might provide similar functionality? > > On Mon, Feb 9, 2015 at 6:47 PM, Nicholas Chammas > wrote: >> +1 to an "official" deprecation + redirecting users to some other >> +project >> that will or already is taking this on. >> >> Nate? >> >> >> >> On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell >> wrote: >>> >>> I have wondered whether we should sort of deprecated it more >>> officially, since otherwise I think people have the reasonable >>> expectation based on the current code that Spark intends to support >>> "complete" Debian packaging as part of the upstream build. Having >>> something that's sort-of maintained but no one is helping review and >>> merge patches on it or make it fully functional, IMO that doesn't >>> benefit us or our users. There are a bunch of other projects that are >>> specifically devoted to packaging, so it seems like there is a clear >>> separation of concerns here. >>> >>> On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra >>> >>> wrote: >>> >> >>> >> it sounds like nobody intends these to be used to actually deploy >>> >> Spark >>> > >>> > >>> > I wouldn't go quite that far. What we have now can serve as useful >>> > input to a deployment tool like Chef, but the user is then going to >>> > need to add some customization or configuration within the context >>> > of that tooling to get Spark installed just the way they want. So >>> > it is not so much that the current Debian packaging can't be used >>> > as that it has never really been intended to be a completely >>> > finished product that a newcomer could, for example, use to install >>> > Spark completely and quickly to Ubuntu and have a fully-functional >>> > environment in which they could then run all of the examples, >>> > tutorials, etc. >>> > >>> > Getting to that level of packaging (and maintenance) is something >>> > that I'm not sure we want to do since that is a better fit with >>> > Bigtop and the efforts of Cloudera, Horton Works, MapR, etc. to >>> > distribute Spark. >>> > >>> > On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen wrote: >>> > >>> >> This is a straw poll to assess whether there is support to keep >>> >> and fix, or remove, the Debian packaging-related config in Spark. >>> >> >>> >> I see several oldish outstanding JIRAs relating to problems in the >>> >> packaging: >>> >> >>> >> https://issues.apache.org/jira/browse/SPARK-1799 >>> >> https://issues.apache.org/jira/browse/SPARK-2614 >>> >> https://issues.apache.org/jira/browse/SPARK-3624 >>> >> https://issues.apache.org/jira/browse/SPARK-4436 >>> >> (and a similar idea about making RPMs) >>> >> https://issues.apache.org/jira/browse/SPARK-665 >>> >> >>> >> The original motivation seems related to Chef: >
RE: Keep or remove Debian packaging in Spark?
This could be something if the spark community wanted to not maintain debs/rpms directly via the project could direct interested efforts towards apache bigtop. Right now debs/rpms of bigtop components, as well as related tests is a focus. Something that would be great is if at least one spark committer with interests in config/pkg/testing could be liason and pt for bigtop efforts. Right now focus on bigtop 0.9, which currently includes spark 1.2. Jira for items included in 0.9 can be found here: https://issues.apache.org/jira/browse/BIGTOP-1480 -Original Message- From: Sean Owen [mailto:so...@cloudera.com] Sent: Monday, February 9, 2015 3:52 PM To: Nicholas Chammas Cc: Patrick Wendell; Mark Hamstra; dev Subject: Re: Keep or remove Debian packaging in Spark? What about this straw man proposal: deprecate in 1.3 with some kind of message in the build, and remove for 1.4? And add a pointer to any third-party packaging that might provide similar functionality? On Mon, Feb 9, 2015 at 6:47 PM, Nicholas Chammas wrote: > +1 to an "official" deprecation + redirecting users to some other > +project > that will or already is taking this on. > > Nate? > > > > On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell > wrote: >> >> I have wondered whether we should sort of deprecated it more >> officially, since otherwise I think people have the reasonable >> expectation based on the current code that Spark intends to support >> "complete" Debian packaging as part of the upstream build. Having >> something that's sort-of maintained but no one is helping review and >> merge patches on it or make it fully functional, IMO that doesn't >> benefit us or our users. There are a bunch of other projects that are >> specifically devoted to packaging, so it seems like there is a clear >> separation of concerns here. >> >> On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra >> >> wrote: >> >> >> >> it sounds like nobody intends these to be used to actually deploy >> >> Spark >> > >> > >> > I wouldn't go quite that far. What we have now can serve as useful >> > input to a deployment tool like Chef, but the user is then going to >> > need to add some customization or configuration within the context >> > of that tooling to get Spark installed just the way they want. So >> > it is not so much that the current Debian packaging can't be used >> > as that it has never really been intended to be a completely >> > finished product that a newcomer could, for example, use to install >> > Spark completely and quickly to Ubuntu and have a fully-functional >> > environment in which they could then run all of the examples, >> > tutorials, etc. >> > >> > Getting to that level of packaging (and maintenance) is something >> > that I'm not sure we want to do since that is a better fit with >> > Bigtop and the efforts of Cloudera, Horton Works, MapR, etc. to >> > distribute Spark. >> > >> > On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen wrote: >> > >> >> This is a straw poll to assess whether there is support to keep >> >> and fix, or remove, the Debian packaging-related config in Spark. >> >> >> >> I see several oldish outstanding JIRAs relating to problems in the >> >> packaging: >> >> >> >> https://issues.apache.org/jira/browse/SPARK-1799 >> >> https://issues.apache.org/jira/browse/SPARK-2614 >> >> https://issues.apache.org/jira/browse/SPARK-3624 >> >> https://issues.apache.org/jira/browse/SPARK-4436 >> >> (and a similar idea about making RPMs) >> >> https://issues.apache.org/jira/browse/SPARK-665 >> >> >> >> The original motivation seems related to Chef: >> >> >> >> >> >> >> >> https://issues.apache.org/jira/browse/SPARK-2614?focusedCommentId= >> >> 14070908&page=com.atlassian.jira.plugin.system.issuetabpanels:comm >> >> ent-tabpanel#comment-14070908 >> >> >> >> Mark's recent comments cast some doubt on whether it is essential: >> >> >> >> https://github.com/apache/spark/pull/4277#issuecomment-72114226 >> >> >> >> and in recent conversations I didn't hear dissent to the idea of >> >> removing this. >> >> >> >> Is this still useful enough to fix up? All else equal I'd like to >> >> start to walk back some of the complexity of the build, but I >> >> don
Re: Keep or remove Debian packaging in Spark?
What about this straw man proposal: deprecate in 1.3 with some kind of message in the build, and remove for 1.4? And add a pointer to any third-party packaging that might provide similar functionality? On Mon, Feb 9, 2015 at 6:47 PM, Nicholas Chammas wrote: > +1 to an "official" deprecation + redirecting users to some other project > that will or already is taking this on. > > Nate? > > > > On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell > wrote: >> >> I have wondered whether we should sort of deprecated it more >> officially, since otherwise I think people have the reasonable >> expectation based on the current code that Spark intends to support >> "complete" Debian packaging as part of the upstream build. Having >> something that's sort-of maintained but no one is helping review and >> merge patches on it or make it fully functional, IMO that doesn't >> benefit us or our users. There are a bunch of other projects that are >> specifically devoted to packaging, so it seems like there is a clear >> separation of concerns here. >> >> On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra >> wrote: >> >> >> >> it sounds like nobody intends these to be used to actually deploy Spark >> > >> > >> > I wouldn't go quite that far. What we have now can serve as useful >> > input >> > to a deployment tool like Chef, but the user is then going to need to >> > add >> > some customization or configuration within the context of that tooling >> > to >> > get Spark installed just the way they want. So it is not so much that >> > the >> > current Debian packaging can't be used as that it has never really been >> > intended to be a completely finished product that a newcomer could, for >> > example, use to install Spark completely and quickly to Ubuntu and have >> > a >> > fully-functional environment in which they could then run all of the >> > examples, tutorials, etc. >> > >> > Getting to that level of packaging (and maintenance) is something that >> > I'm >> > not sure we want to do since that is a better fit with Bigtop and the >> > efforts of Cloudera, Horton Works, MapR, etc. to distribute Spark. >> > >> > On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen wrote: >> > >> >> This is a straw poll to assess whether there is support to keep and >> >> fix, or remove, the Debian packaging-related config in Spark. >> >> >> >> I see several oldish outstanding JIRAs relating to problems in the >> >> packaging: >> >> >> >> https://issues.apache.org/jira/browse/SPARK-1799 >> >> https://issues.apache.org/jira/browse/SPARK-2614 >> >> https://issues.apache.org/jira/browse/SPARK-3624 >> >> https://issues.apache.org/jira/browse/SPARK-4436 >> >> (and a similar idea about making RPMs) >> >> https://issues.apache.org/jira/browse/SPARK-665 >> >> >> >> The original motivation seems related to Chef: >> >> >> >> >> >> >> >> https://issues.apache.org/jira/browse/SPARK-2614?focusedCommentId=14070908&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14070908 >> >> >> >> Mark's recent comments cast some doubt on whether it is essential: >> >> >> >> https://github.com/apache/spark/pull/4277#issuecomment-72114226 >> >> >> >> and in recent conversations I didn't hear dissent to the idea of >> >> removing >> >> this. >> >> >> >> Is this still useful enough to fix up? All else equal I'd like to >> >> start to walk back some of the complexity of the build, but I don't >> >> know how all-else-equal it is. Certainly, it sounds like nobody >> >> intends these to be used to actually deploy Spark. >> >> >> >> I don't doubt it's useful to someone, but can they maintain the >> >> packaging logic elsewhere? >> >> >> >> - >> >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> >> >> >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> > - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Keep or remove Debian packaging in Spark?
+1 to an "official" deprecation + redirecting users to some other project that will or already is taking this on. Nate? On Mon Feb 09 2015 at 10:08:27 AM Patrick Wendell wrote: > I have wondered whether we should sort of deprecated it more > officially, since otherwise I think people have the reasonable > expectation based on the current code that Spark intends to support > "complete" Debian packaging as part of the upstream build. Having > something that's sort-of maintained but no one is helping review and > merge patches on it or make it fully functional, IMO that doesn't > benefit us or our users. There are a bunch of other projects that are > specifically devoted to packaging, so it seems like there is a clear > separation of concerns here. > > On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra > wrote: > >> > >> it sounds like nobody intends these to be used to actually deploy Spark > > > > > > I wouldn't go quite that far. What we have now can serve as useful input > > to a deployment tool like Chef, but the user is then going to need to add > > some customization or configuration within the context of that tooling to > > get Spark installed just the way they want. So it is not so much that > the > > current Debian packaging can't be used as that it has never really been > > intended to be a completely finished product that a newcomer could, for > > example, use to install Spark completely and quickly to Ubuntu and have a > > fully-functional environment in which they could then run all of the > > examples, tutorials, etc. > > > > Getting to that level of packaging (and maintenance) is something that > I'm > > not sure we want to do since that is a better fit with Bigtop and the > > efforts of Cloudera, Horton Works, MapR, etc. to distribute Spark. > > > > On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen wrote: > > > >> This is a straw poll to assess whether there is support to keep and > >> fix, or remove, the Debian packaging-related config in Spark. > >> > >> I see several oldish outstanding JIRAs relating to problems in the > >> packaging: > >> > >> https://issues.apache.org/jira/browse/SPARK-1799 > >> https://issues.apache.org/jira/browse/SPARK-2614 > >> https://issues.apache.org/jira/browse/SPARK-3624 > >> https://issues.apache.org/jira/browse/SPARK-4436 > >> (and a similar idea about making RPMs) > >> https://issues.apache.org/jira/browse/SPARK-665 > >> > >> The original motivation seems related to Chef: > >> > >> > >> https://issues.apache.org/jira/browse/SPARK-2614?focusedComm > entId=14070908&page=com.atlassian.jira.plugin.system.issuetabpanels: > comment-tabpanel#comment-14070908 > >> > >> Mark's recent comments cast some doubt on whether it is essential: > >> > >> https://github.com/apache/spark/pull/4277#issuecomment-72114226 > >> > >> and in recent conversations I didn't hear dissent to the idea of > removing > >> this. > >> > >> Is this still useful enough to fix up? All else equal I'd like to > >> start to walk back some of the complexity of the build, but I don't > >> know how all-else-equal it is. Certainly, it sounds like nobody > >> intends these to be used to actually deploy Spark. > >> > >> I don't doubt it's useful to someone, but can they maintain the > >> packaging logic elsewhere? > >> > >> - > >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > >> For additional commands, e-mail: dev-h...@spark.apache.org > >> > >> > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
Re: Keep or remove Debian packaging in Spark?
I have wondered whether we should sort of deprecated it more officially, since otherwise I think people have the reasonable expectation based on the current code that Spark intends to support "complete" Debian packaging as part of the upstream build. Having something that's sort-of maintained but no one is helping review and merge patches on it or make it fully functional, IMO that doesn't benefit us or our users. There are a bunch of other projects that are specifically devoted to packaging, so it seems like there is a clear separation of concerns here. On Mon, Feb 9, 2015 at 7:31 AM, Mark Hamstra wrote: >> >> it sounds like nobody intends these to be used to actually deploy Spark > > > I wouldn't go quite that far. What we have now can serve as useful input > to a deployment tool like Chef, but the user is then going to need to add > some customization or configuration within the context of that tooling to > get Spark installed just the way they want. So it is not so much that the > current Debian packaging can't be used as that it has never really been > intended to be a completely finished product that a newcomer could, for > example, use to install Spark completely and quickly to Ubuntu and have a > fully-functional environment in which they could then run all of the > examples, tutorials, etc. > > Getting to that level of packaging (and maintenance) is something that I'm > not sure we want to do since that is a better fit with Bigtop and the > efforts of Cloudera, Horton Works, MapR, etc. to distribute Spark. > > On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen wrote: > >> This is a straw poll to assess whether there is support to keep and >> fix, or remove, the Debian packaging-related config in Spark. >> >> I see several oldish outstanding JIRAs relating to problems in the >> packaging: >> >> https://issues.apache.org/jira/browse/SPARK-1799 >> https://issues.apache.org/jira/browse/SPARK-2614 >> https://issues.apache.org/jira/browse/SPARK-3624 >> https://issues.apache.org/jira/browse/SPARK-4436 >> (and a similar idea about making RPMs) >> https://issues.apache.org/jira/browse/SPARK-665 >> >> The original motivation seems related to Chef: >> >> >> https://issues.apache.org/jira/browse/SPARK-2614?focusedCommentId=14070908&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14070908 >> >> Mark's recent comments cast some doubt on whether it is essential: >> >> https://github.com/apache/spark/pull/4277#issuecomment-72114226 >> >> and in recent conversations I didn't hear dissent to the idea of removing >> this. >> >> Is this still useful enough to fix up? All else equal I'd like to >> start to walk back some of the complexity of the build, but I don't >> know how all-else-equal it is. Certainly, it sounds like nobody >> intends these to be used to actually deploy Spark. >> >> I don't doubt it's useful to someone, but can they maintain the >> packaging logic elsewhere? >> >> - >> To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org >> For additional commands, e-mail: dev-h...@spark.apache.org >> >> - To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org For additional commands, e-mail: dev-h...@spark.apache.org
Re: Keep or remove Debian packaging in Spark?
> > it sounds like nobody intends these to be used to actually deploy Spark I wouldn't go quite that far. What we have now can serve as useful input to a deployment tool like Chef, but the user is then going to need to add some customization or configuration within the context of that tooling to get Spark installed just the way they want. So it is not so much that the current Debian packaging can't be used as that it has never really been intended to be a completely finished product that a newcomer could, for example, use to install Spark completely and quickly to Ubuntu and have a fully-functional environment in which they could then run all of the examples, tutorials, etc. Getting to that level of packaging (and maintenance) is something that I'm not sure we want to do since that is a better fit with Bigtop and the efforts of Cloudera, Horton Works, MapR, etc. to distribute Spark. On Mon, Feb 9, 2015 at 2:41 AM, Sean Owen wrote: > This is a straw poll to assess whether there is support to keep and > fix, or remove, the Debian packaging-related config in Spark. > > I see several oldish outstanding JIRAs relating to problems in the > packaging: > > https://issues.apache.org/jira/browse/SPARK-1799 > https://issues.apache.org/jira/browse/SPARK-2614 > https://issues.apache.org/jira/browse/SPARK-3624 > https://issues.apache.org/jira/browse/SPARK-4436 > (and a similar idea about making RPMs) > https://issues.apache.org/jira/browse/SPARK-665 > > The original motivation seems related to Chef: > > > https://issues.apache.org/jira/browse/SPARK-2614?focusedCommentId=14070908&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14070908 > > Mark's recent comments cast some doubt on whether it is essential: > > https://github.com/apache/spark/pull/4277#issuecomment-72114226 > > and in recent conversations I didn't hear dissent to the idea of removing > this. > > Is this still useful enough to fix up? All else equal I'd like to > start to walk back some of the complexity of the build, but I don't > know how all-else-equal it is. Certainly, it sounds like nobody > intends these to be used to actually deploy Spark. > > I don't doubt it's useful to someone, but can they maintain the > packaging logic elsewhere? > > - > To unsubscribe, e-mail: dev-unsubscr...@spark.apache.org > For additional commands, e-mail: dev-h...@spark.apache.org > >
