On Wed, Sep 25, 2013 at 01:26PM, Shane Huang wrote: > I think it's good to have Bigtop to package Spark. But in this track we're > just targeting enhancing the usability of Spark itself without Bigtop. > After all, few of our customers used Bigtop.
Bigtop - in this particular application - is just a way of building packages. But what I am saying is that Bigtop has this structure in place already - it can be just copied. Cos > On Wed, Sep 25, 2013 at 1:20 PM, Shane Huang <[email protected]>wrote: > > > I think it's good to have Bigtop to package Spark. But I > > > > > > On Wed, Sep 25, 2013 at 1:16 PM, Konstantin Boudnik <[email protected]>wrote: > > > >> Late to the game, but... Bigtop is packaging Spark now as a part of the > >> standard distribution - our release 0.7.0 is around the corner. And we do > >> it > >> in the same way that has been done for Hadoop. Perhaps it worth looking > >> into... > >> > >> Lemme know if you have any questions, > >> Cos > >> > >> On Sun, Sep 22, 2013 at 12:07PM, Shane Huang wrote: > >> > And I created a new issue SPARK-915 to track the re-org of scripts as > >> > SPARK-544 only talks about Config. > >> > https://spark-project.atlassian.net/browse/SPARK-915 > >> > > >> > > >> > On Wed, Sep 18, 2013 at 1:42 AM, Matei Zaharia <[email protected] > >> >wrote: > >> > > >> > > Hi Shane, > >> > > > >> > > I agree with all these points. Improving the configuration system is > >> one > >> > > of the main things I'd like to have in the next release. > >> > > > >> > > > 1) Usually the application developers/users and platform > >> administrators > >> > > > belongs to two teams. So it's better to separate the scripts used by > >> > > > administrators and application users, e.g. put them in sbin and bin > >> > > folders > >> > > > respectively > >> > > > >> > > Yup, right now we don't have any attempt to install on standard system > >> > > paths. > >> > > > >> > > > 3) If there are multiple ways to specify an option, an overriding > >> rule > >> > > > should be present and should not be error-prone. > >> > > > >> > > Yes, I think this should always be Configuration class in code > > >> system > >> > > properties > env vars. Over time we will deprecate the env vars and > >> maybe > >> > > even system properties. > >> > > > >> > > > 4) Currently the options are set and get using System property. > >> It's hard > >> > > > to manage and inconvenient for users. It's good to gather the > >> options > >> > > into > >> > > > one file using format like xml or json. > >> > > > >> > > I think this is the main thing to do first -- pick one configuration > >> class > >> > > and change the code to use this. > >> > > > >> > > > Our rough proposal: > >> > > > > >> > > > - Scripts > >> > > > > >> > > > 1. make an "sbin" folder containing all the scripts for > >> administrators, > >> > > > specifically, > >> > > > - all service administration scripts, i.e. start-*, stop-*, > >> > > > slaves.sh, *-daemons, *-daemon scripts > >> > > > - low-level or internally used utility scripts, i.e. > >> > > > compute-classpath, spark-config, spark-class, spark-executor > >> > > > 2. make a "bin" folder containing all the scripts for application > >> > > > developers/users, specifically, > >> > > > - user level app running scripts, i.e. pyspark, spark-shell, > >> and we > >> > > > propose to add a script "spark" for users to run applications > >> (very > >> > > much > >> > > > like spark-class but may add some more control or convenient > >> > > utilities) > >> > > > - scripts for status checking, e.g. spark and hadoop version > >> > > > checking, running applications checking, etc. We can make this > >> a > >> > > separate > >> > > > script or add functionality to "spark" script. > >> > > > 3. No wandering scripts outside the sbin and bin folders > >> > > > >> > > Makes sense. > >> > > > >> > > > - Configurations/Options and overriding rule > >> > > > > >> > > > 1. Define a Configuration class which contains all the options > >> > > available > >> > > > for Spark application. A Configuration instance can be > >> de-/serialized > >> > > > from/to a json formatted file. > >> > > > 2. Each application (SparkContext) has one Configuration instance > >> and > >> > > it > >> > > > is initialized by the application which creates it (either read > >> from > >> > > file > >> > > > or passed from command line options or env SPARK_JAVA_OPTS). > >> > > > 3. When launching an Executor on a node, the Configuration is > >> firstly > >> > > > initialized using the node-local configuration file as default. > >> The > >> > > > Configuration passed from application driver context will > >> override any > >> > > > options specified in default. > >> > > > >> > > This sounds great to me! The one thing I'll add is that we might want > >> to > >> > > prevent applications from overriding certain settings on each node, > >> such as > >> > > work directories. The best way is to probably just ignore the app's > >> version > >> > > of those settings in the Executor. > >> > > > >> > > If you guys would like, feel free to write up this design on > >> SPARK-544 and > >> > > start working on it. I think it looks good. > >> > > > >> > > Matei > >> > > >> > > >> > > >> > > >> > -- > >> > *Shane Huang * > >> > *Intel Asia-Pacific R&D Ltd.* > >> > *Email: [email protected]* > >> > > > > > > > > -- > > *Shane Huang * > > *Intel Asia-Pacific R&D Ltd.* > > *Email: [email protected]* > > > > > > > -- > *Shane Huang * > *Intel Asia-Pacific R&D Ltd.* > *Email: [email protected]*
signature.asc
Description: Digital signature
