Hi Chan Shane and I happened to try to contribute configure feature to spark, could we cooperate to implement it?
-----Original Message----- From: Evan Chan [mailto:[email protected]] Sent: Thursday, September 26, 2013 2:43 PM To: [email protected] Subject: Re: Propose to Re-organize the scripts and configurations Shane, and others, Let's work together on the configuration thing. I had proposed in a separate thread to use Typesafe Config to hold all configuration (essentially a configuration class, but which can read from both JSON files as well as -D java command line args). Typesafe Config works much much better than a simple config class, and also better than Hadoop configs. It also has advantages over JSON (more readable, comments). It would also be the easiest to transition from the current scheme, since the current java system properties can be seamlessly integrated. I would be happy to contribute this back soon because it is also a big pain point for us. I also have extensive experience with both Typesafe Config and other config systems. I would definitely start with SparkContext and work our way out from there. In fact I can submit a patch for everyone to test out fairly quickly just for SparkContext. -Evan On Tue, Sep 24, 2013 at 10:26 PM, Shane Huang <[email protected]>wrote: > I think it's good to have Bigtop to package Spark. But in this track > we're just targeting enhancing the usability of Spark itself without Bigtop. > After all, few of our customers used Bigtop. > > > On Wed, Sep 25, 2013 at 1:20 PM, Shane Huang <[email protected] > >wrote: > > > I think it's good to have Bigtop to package Spark. But I > > > > > > On Wed, Sep 25, 2013 at 1:16 PM, Konstantin Boudnik <[email protected] > >wrote: > > > >> Late to the game, but... Bigtop is packaging Spark now as a part of > >> the standard distribution - our release 0.7.0 is around the corner. > >> And we > do > >> it > >> in the same way that has been done for Hadoop. Perhaps it worth > >> looking into... > >> > >> Lemme know if you have any questions, > >> Cos > >> > >> On Sun, Sep 22, 2013 at 12:07PM, Shane Huang wrote: > >> > And I created a new issue SPARK-915 to track the re-org of > >> > scripts as > >> > SPARK-544 only talks about Config. > >> > https://spark-project.atlassian.net/browse/SPARK-915 > >> > > >> > > >> > On Wed, Sep 18, 2013 at 1:42 AM, Matei Zaharia < > [email protected] > >> >wrote: > >> > > >> > > Hi Shane, > >> > > > >> > > I agree with all these points. Improving the configuration > >> > > system is > >> one > >> > > of the main things I'd like to have in the next release. > >> > > > >> > > > 1) Usually the application developers/users and platform > >> administrators > >> > > > belongs to two teams. So it's better to separate the scripts > >> > > > used > by > >> > > > administrators and application users, e.g. put them in sbin > >> > > > and > bin > >> > > folders > >> > > > respectively > >> > > > >> > > Yup, right now we don't have any attempt to install on standard > system > >> > > paths. > >> > > > >> > > > 3) If there are multiple ways to specify an option, an > >> > > > overriding > >> rule > >> > > > should be present and should not be error-prone. > >> > > > >> > > Yes, I think this should always be Configuration class in code > >> > > > > >> system > >> > > properties > env vars. Over time we will deprecate the env vars > >> > > and > >> maybe > >> > > even system properties. > >> > > > >> > > > 4) Currently the options are set and get using System property. > >> It's hard > >> > > > to manage and inconvenient for users. It's good to gather the > >> options > >> > > into > >> > > > one file using format like xml or json. > >> > > > >> > > I think this is the main thing to do first -- pick one > >> > > configuration > >> class > >> > > and change the code to use this. > >> > > > >> > > > Our rough proposal: > >> > > > > >> > > > - Scripts > >> > > > > >> > > > 1. make an "sbin" folder containing all the scripts for > >> administrators, > >> > > > specifically, > >> > > > - all service administration scripts, i.e. start-*, stop-*, > >> > > > slaves.sh, *-daemons, *-daemon scripts > >> > > > - low-level or internally used utility scripts, i.e. > >> > > > compute-classpath, spark-config, spark-class, spark-executor > >> > > > 2. make a "bin" folder containing all the scripts for > application > >> > > > developers/users, specifically, > >> > > > - user level app running scripts, i.e. pyspark, > >> > > > spark-shell, > >> and we > >> > > > propose to add a script "spark" for users to run > >> > > > applications > >> (very > >> > > much > >> > > > like spark-class but may add some more control or > >> > > > convenient > >> > > utilities) > >> > > > - scripts for status checking, e.g. spark and hadoop version > >> > > > checking, running applications checking, etc. We can > >> > > > make > this > >> a > >> > > separate > >> > > > script or add functionality to "spark" script. > >> > > > 3. No wandering scripts outside the sbin and bin folders > >> > > > >> > > Makes sense. > >> > > > >> > > > - Configurations/Options and overriding rule > >> > > > > >> > > > 1. Define a Configuration class which contains all the > >> > > > options > >> > > available > >> > > > for Spark application. A Configuration instance can be > >> de-/serialized > >> > > > from/to a json formatted file. > >> > > > 2. Each application (SparkContext) has one Configuration > instance > >> and > >> > > it > >> > > > is initialized by the application which creates it (either > >> > > > read > >> from > >> > > file > >> > > > or passed from command line options or env SPARK_JAVA_OPTS). > >> > > > 3. When launching an Executor on a node, the Configuration > >> > > > is > >> firstly > >> > > > initialized using the node-local configuration file as default. > >> The > >> > > > Configuration passed from application driver context will > >> override any > >> > > > options specified in default. > >> > > > >> > > This sounds great to me! The one thing I'll add is that we > >> > > might > want > >> to > >> > > prevent applications from overriding certain settings on each > >> > > node, > >> such as > >> > > work directories. The best way is to probably just ignore the > >> > > app's > >> version > >> > > of those settings in the Executor. > >> > > > >> > > If you guys would like, feel free to write up this design on > >> SPARK-544 and > >> > > start working on it. I think it looks good. > >> > > > >> > > Matei > >> > > >> > > >> > > >> > > >> > -- > >> > *Shane Huang * > >> > *Intel Asia-Pacific R&D Ltd.* > >> > *Email: [email protected]* > >> > > > > > > > > -- > > *Shane Huang * > > *Intel Asia-Pacific R&D Ltd.* > > *Email: [email protected]* > > > > > > > -- > *Shane Huang * > *Intel Asia-Pacific R&D Ltd.* > *Email: [email protected]* > -- -- Evan Chan Staff Engineer [email protected] | <http://www.ooyala.com/> <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
