[
https://issues.apache.org/jira/browse/SPARK-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Apache Spark reassigned SPARK-6214:
-----------------------------------
Assignee: Apache Spark
> Allow configuration options to use a simple expression language
> ---------------------------------------------------------------
>
> Key: SPARK-6214
> URL: https://issues.apache.org/jira/browse/SPARK-6214
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Reporter: Dale Richardson
> Assignee: Apache Spark
> Priority: Minor
>
> This is a proposal to allow for configuration options to be specified via a
> simple expression language. This language would have the following features:
> * Allow for basic arithmetic (+-/*) with support for bracketed expressions
> and standard precedence rules.
> * Support for and normalisation of common units of scale eg. MB, GB,
> seconds,minutes,hours, days and weeks.
> * Allow for the referencing of basic environmental information currently
> defined as:
> numCores: Number of cores assigned to the JVM
> physicalMemoryBytes: Memory size of hosting machine
> JVMTotalMemoryBytes: current bytes of memory allocated to the JVM
> JVMMaxMemoryBytes: Maximum number of bytes of memory available to the JVM
> JVMFreeMemoryBytes: maxMemoryBytes - totalMemoryBytes
> * Allow for the limited referencing of other configuration values when
> specifying values. (Other configuration values must be initialised and
> explicitly passed into the expression evaluator for this functionality to be
> enabled).
>
> Such a feature would have the following end-user benefits:
> * Allow for the flexibility in specifying time intervals or byte quantities
> in appropriate and easy to follow units e.g. 1 week rather rather then
> 604800 seconds
> * Have a consistent means of entering configuration information regardless of
> the configuration option being added. (eg questions such as ‘is the
> particular option specified in ms or seconds?’ become irrelevant, because the
> user can pick what ever unit makes sense for the magnitude of the value they
> are specifying)
> * Allow for the scaling of a configuration option in relation to a system
> attributes. e.g.
> --cores "numCores - 1"
> spark.executor.memory = physicalMemoryBytes - 1.5 GB
> * Being able to scale multiple configuration options together eg:
> spark.driver.memory = 0.75 * physicalMemoryBytes
> spark.driver.maxResultSize = spark.driver.memory * 0.8
> A safety features has been added so that the expression evaluator is only
> used if the configuration strings contains a magic character (currently '!')
> as the first character.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
