+1 for Typesafe configFactory and config Sent from my iPad
On Sep 25, 2013, at 11:42 PM, Evan Chan <[email protected]> wrote: > Shane, and others, > > Let's work together on the configuration thing. I had proposed in a > separate thread to use Typesafe Config to hold all configuration > (essentially a configuration class, but which can read from both JSON files > as well as -D java command line args). > > Typesafe Config works much much better than a simple config class, and also > better than Hadoop configs. It also has advantages over JSON (more > readable, comments). It would also be the easiest to transition from the > current scheme, since the current java system properties can be seamlessly > integrated. > > I would be happy to contribute this back soon because it is also a big pain > point for us. I also have extensive experience with both Typesafe Config > and other config systems. > > I would definitely start with SparkContext and work our way out from there. > In fact I can submit a patch for everyone to test out fairly quickly > just for SparkContext. > > -Evan > > > > On Tue, Sep 24, 2013 at 10:26 PM, Shane Huang <[email protected]>wrote: > >> I think it's good to have Bigtop to package Spark. But in this track we're >> just targeting enhancing the usability of Spark itself without Bigtop. >> After all, few of our customers used Bigtop. >> >> >> On Wed, Sep 25, 2013 at 1:20 PM, Shane Huang <[email protected] >>> wrote: >> >>> I think it's good to have Bigtop to package Spark. But I >>> >>> >>> On Wed, Sep 25, 2013 at 1:16 PM, Konstantin Boudnik <[email protected] >>> wrote: >>> >>>> Late to the game, but... Bigtop is packaging Spark now as a part of the >>>> standard distribution - our release 0.7.0 is around the corner. And we >> do >>>> it >>>> in the same way that has been done for Hadoop. Perhaps it worth looking >>>> into... >>>> >>>> Lemme know if you have any questions, >>>> Cos >>>> >>>> On Sun, Sep 22, 2013 at 12:07PM, Shane Huang wrote: >>>>> And I created a new issue SPARK-915 to track the re-org of scripts as >>>>> SPARK-544 only talks about Config. >>>>> https://spark-project.atlassian.net/browse/SPARK-915 >>>>> >>>>> >>>>> On Wed, Sep 18, 2013 at 1:42 AM, Matei Zaharia < >> [email protected] >>>>> wrote: >>>>> >>>>>> Hi Shane, >>>>>> >>>>>> I agree with all these points. Improving the configuration system is >>>> one >>>>>> of the main things I'd like to have in the next release. >>>>>> >>>>>>> 1) Usually the application developers/users and platform >>>> administrators >>>>>>> belongs to two teams. So it's better to separate the scripts used >> by >>>>>>> administrators and application users, e.g. put them in sbin and >> bin >>>>>> folders >>>>>>> respectively >>>>>> >>>>>> Yup, right now we don't have any attempt to install on standard >> system >>>>>> paths. >>>>>> >>>>>>> 3) If there are multiple ways to specify an option, an overriding >>>> rule >>>>>>> should be present and should not be error-prone. >>>>>> >>>>>> Yes, I think this should always be Configuration class in code > >>>> system >>>>>> properties > env vars. Over time we will deprecate the env vars and >>>> maybe >>>>>> even system properties. >>>>>> >>>>>>> 4) Currently the options are set and get using System property. >>>> It's hard >>>>>>> to manage and inconvenient for users. It's good to gather the >>>> options >>>>>> into >>>>>>> one file using format like xml or json. >>>>>> >>>>>> I think this is the main thing to do first -- pick one configuration >>>> class >>>>>> and change the code to use this. >>>>>> >>>>>>> Our rough proposal: >>>>>>> >>>>>>> - Scripts >>>>>>> >>>>>>> 1. make an "sbin" folder containing all the scripts for >>>> administrators, >>>>>>> specifically, >>>>>>> - all service administration scripts, i.e. start-*, stop-*, >>>>>>> slaves.sh, *-daemons, *-daemon scripts >>>>>>> - low-level or internally used utility scripts, i.e. >>>>>>> compute-classpath, spark-config, spark-class, spark-executor >>>>>>> 2. make a "bin" folder containing all the scripts for >> application >>>>>>> developers/users, specifically, >>>>>>> - user level app running scripts, i.e. pyspark, spark-shell, >>>> and we >>>>>>> propose to add a script "spark" for users to run applications >>>> (very >>>>>> much >>>>>>> like spark-class but may add some more control or convenient >>>>>> utilities) >>>>>>> - scripts for status checking, e.g. spark and hadoop version >>>>>>> checking, running applications checking, etc. We can make >> this >>>> a >>>>>> separate >>>>>>> script or add functionality to "spark" script. >>>>>>> 3. No wandering scripts outside the sbin and bin folders >>>>>> >>>>>> Makes sense. >>>>>> >>>>>>> - Configurations/Options and overriding rule >>>>>>> >>>>>>> 1. Define a Configuration class which contains all the options >>>>>> available >>>>>>> for Spark application. A Configuration instance can be >>>> de-/serialized >>>>>>> from/to a json formatted file. >>>>>>> 2. Each application (SparkContext) has one Configuration >> instance >>>> and >>>>>> it >>>>>>> is initialized by the application which creates it (either read >>>> from >>>>>> file >>>>>>> or passed from command line options or env SPARK_JAVA_OPTS). >>>>>>> 3. When launching an Executor on a node, the Configuration is >>>> firstly >>>>>>> initialized using the node-local configuration file as default. >>>> The >>>>>>> Configuration passed from application driver context will >>>> override any >>>>>>> options specified in default. >>>>>> >>>>>> This sounds great to me! The one thing I'll add is that we might >> want >>>> to >>>>>> prevent applications from overriding certain settings on each node, >>>> such as >>>>>> work directories. The best way is to probably just ignore the app's >>>> version >>>>>> of those settings in the Executor. >>>>>> >>>>>> If you guys would like, feel free to write up this design on >>>> SPARK-544 and >>>>>> start working on it. I think it looks good. >>>>>> >>>>>> Matei >>>>> >>>>> >>>>> >>>>> >>>>> -- >>>>> *Shane Huang * >>>>> *Intel Asia-Pacific R&D Ltd.* >>>>> *Email: [email protected]* >>>> >>> >>> >>> >>> -- >>> *Shane Huang * >>> *Intel Asia-Pacific R&D Ltd.* >>> *Email: [email protected]* >>> >>> >> >> >> -- >> *Shane Huang * >> *Intel Asia-Pacific R&D Ltd.* >> *Email: [email protected]* >> > > > > -- > -- > Evan Chan > Staff Engineer > [email protected] | > > <http://www.ooyala.com/> > <http://www.facebook.com/ooyala><http://www.linkedin.com/company/ooyala><http://www.twitter.com/ooyala>
