I like the idea of using Typesafe Config. Nick, we'd be glad to work with you after we gathered enough opinions and come to a consensus of the approach.
On 2013-9-16, at 16:52, Nick Pentreath <[email protected]> wrote: > There was another discussion on the old dev list about this: > https://groups.google.com/forum/#!msg/spark-developers/GL2_DwAeh5s/9rwQ3iDa2t4J > > I tend to agree with having configuration sitting in JSON (or properties > files) and using the Typesafe Config library which can parse both. > > Something I've used in my apps is along these lines: > https://gist.github.com/MLnick/6578146 > > It's then easy to have default config overridden with CLI for example: > val conf = cliConf.withFallback(defaultConf) > > I'd be happy to be involved in working on this if there is a consensus > about best approach > > N > > > > > > On Mon, Sep 16, 2013 at 9:29 AM, Mike <[email protected]> wrote: > >> Shane Huang wrote: >>> we found the current organization of the scripts and configuration a >>> bit confusing and inconvenient >> >> ditto >> >>> - Scripts >> >> I wonder why the work of these scripts wasn't mostly done in Scala. >> Seems roundabout to use Bash (or Python, in spark-perf) to calculate >> shell environment variables that are then read back into Scala code. >> >>> 1. Define a Configuration class which contains all the options >>> available for Spark application. A Configuration instance can be >>> de-/serialized from/to a json formatted file. >>> 2. Each application (SparkContext) has one Configuration instance and >>> it is initialized by the application which creates it (either read >>> from file or passed from command line options or env SPARK_JAVA_OPTS). >> >> Reminiscent of what Hibernate's been doing for the past decade. Would >> be nice if the Configuration was also exposed through an MBean or such >> so that one can check it's values with certainty. >>
