Dale Richardson created SPARK-6214:
--------------------------------------
Summary: Allow configuration options to use a simple expression
language
Key: SPARK-6214
URL: https://issues.apache.org/jira/browse/SPARK-6214
Project: Spark
Issue Type: Improvement
Components: Spark Core
Reporter: Dale Richardson
Priority: Minor
This is a proposal to allow for configuration options to be specified via a
simple expression language. This language would have the following features:
Allow for basic arithmetic (+-/*) with support bracketed expressions and
standard precedence rules.
Support for and normalisation of common units of reference eg. MB, GB,
seconds,minutes,hours, days and weeks.
Allow for the referencing of basic environmental information currently defined
as:
numCores: Number of cores assigned to the JVM
physicalMemoryBytes:
Memory size of hosting machine
JVMtotalMemoryBytes: current bytes of memory
allocated to the JVM
JVMmaxMemoryBytes: Maximum number of bytes of memory
available to the JVM
JVMfreeMemoryBytes: maxMemoryBytes - totalMemoryBytes
Allow for the limited referencing of other configuration values when specifying
values. (Other configuration values must be initialised and explicitly passed
into the expression evaluator for this functionality to be enabled).
Such a feature would have the following end-user benefits:
Allow for the flexibility in specifying time intervals or byte quantities in
appropriate and easy to follow units e.g. 1 week rather rather then 604800
seconds
Have a consistent means of entering configuration information regardless of the
configuration option being added. (eg questions such as ‘is the particular
option specified in ms or seconds?’ become irrelevant, because the user can
pick what ever unit makes sense for the magnitude of the value they are
specifying)
Allow for the scaling of a configuration option in relation to a system
attributes. e.g.
SPARK_WORKER_CORES = numCores - 1
SPARK_WORKER_MEMORY = physicalMemoryBytes - 1.5 GB
Being able to scale multiple configuration options together eg:
spark.driver.memory = 0.75 * physicalMemoryBytes
spark.driver.maxResultSize = spark.driver.memory * 0.8
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
