Dale Richardson created SPARK-6214: -------------------------------------- Summary: Allow configuration options to use a simple expression language Key: SPARK-6214 URL: https://issues.apache.org/jira/browse/SPARK-6214 Project: Spark Issue Type: Improvement Components: Spark Core Reporter: Dale Richardson Priority: Minor
This is a proposal to allow for configuration options to be specified via a simple expression language. This language would have the following features: Allow for basic arithmetic (+-/*) with support bracketed expressions and standard precedence rules. Support for and normalisation of common units of reference eg. MB, GB, seconds,minutes,hours, days and weeks. Allow for the referencing of basic environmental information currently defined as: numCores: Number of cores assigned to the JVM physicalMemoryBytes: Memory size of hosting machine JVMtotalMemoryBytes: current bytes of memory allocated to the JVM JVMmaxMemoryBytes: Maximum number of bytes of memory available to the JVM JVMfreeMemoryBytes: maxMemoryBytes - totalMemoryBytes Allow for the limited referencing of other configuration values when specifying values. (Other configuration values must be initialised and explicitly passed into the expression evaluator for this functionality to be enabled). Such a feature would have the following end-user benefits: Allow for the flexibility in specifying time intervals or byte quantities in appropriate and easy to follow units e.g. 1 week rather rather then 604800 seconds Have a consistent means of entering configuration information regardless of the configuration option being added. (eg questions such as ‘is the particular option specified in ms or seconds?’ become irrelevant, because the user can pick what ever unit makes sense for the magnitude of the value they are specifying) Allow for the scaling of a configuration option in relation to a system attributes. e.g. SPARK_WORKER_CORES = numCores - 1 SPARK_WORKER_MEMORY = physicalMemoryBytes - 1.5 GB Being able to scale multiple configuration options together eg: spark.driver.memory = 0.75 * physicalMemoryBytes spark.driver.maxResultSize = spark.driver.memory * 0.8 -- This message was sent by Atlassian JIRA (v6.3.4#6332) --------------------------------------------------------------------- To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org For additional commands, e-mail: issues-h...@spark.apache.org