Jacek Lewandowski created SPARK-7169:
----------------------------------------
Summary: Allow to specify metrics configuration more flexibly
Key: SPARK-7169
URL: https://issues.apache.org/jira/browse/SPARK-7169
Project: Spark
Issue Type: Improvement
Components: Spark Core
Affects Versions: 1.3.1, 1.2.2, 1.4.0
Reporter: Jacek Lewandowski
Metrics are configured in {{metrics.properties}} file. Path to this file is
specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
initialisation.
h5.Problem
When the user runs his application he has no way to provide the metrics
configuration for executors. Although one can specify the path to metrics
configuration file (1) the path is common for all the nodes and the client
machine so there is implicit assumption that all the machines has same file in
the same location, and (2) actually the user needs to copy the file manually to
the worker nodes because the file is read before the user files are populated
to the executor local directories. All of this makes it very difficult to play
with the metrics configuration.
h5. Proposed solution
I think that the easiest and the most consistent solution would be to move the
configuration from a separate file directly to {{SparkConf}}. We may prefix all
the configuration settings from the metrics configuration by, say
{{spark.metrics.props}}. For the backward compatibility, these properties would
be loaded from the specified as it works now. Such a solution doesn't change
the API so maybe it could be even included in patch release of Spark 1.2 and
Spark 1.3.
Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
