[
https://issues.apache.org/jira/browse/SPARK-7169?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14518508#comment-14518508
]
Saisai Shao commented on SPARK-7169:
------------------------------------
Hi [~jlewandowski], regard to your second problem, I think you don't have to
copy the metrics configuration file manually to every machine one by one, you
could use spark-submit --file path/to/your/metrics_properties to transfer your
configuration to each executor/container.
And for the first problem, is it a big problem that all the configuration files
need to be in the same directory? I think lot's of Spark as well as Hadoop conf
file has such requirement. But you could configure driver/executor with
different parameters in conf file, since MetricsSystem supports such features.
Yes I think current metrics configuration may not so easy to use, any
improvement is greatly appreciated :).
> Allow to specify metrics configuration more flexibly
> ----------------------------------------------------
>
> Key: SPARK-7169
> URL: https://issues.apache.org/jira/browse/SPARK-7169
> Project: Spark
> Issue Type: Improvement
> Components: Spark Core
> Affects Versions: 1.2.2, 1.3.1
> Reporter: Jacek Lewandowski
> Priority: Minor
>
> Metrics are configured in {{metrics.properties}} file. Path to this file is
> specified in {{SparkConf}} at a key {{spark.metrics.conf}}. The property is
> read when {{MetricsSystem}} is created which means, during {{SparkEnv}}
> initialisation.
> h5.Problem
> When the user runs his application he has no way to provide the metrics
> configuration for executors. Although one can specify the path to metrics
> configuration file (1) the path is common for all the nodes and the client
> machine so there is implicit assumption that all the machines has same file
> in the same location, and (2) actually the user needs to copy the file
> manually to the worker nodes because the file is read before the user files
> are populated to the executor local directories. All of this makes it very
> difficult to play with the metrics configuration.
> h5. Proposed solution
> I think that the easiest and the most consistent solution would be to move
> the configuration from a separate file directly to {{SparkConf}}. We may
> prefix all the configuration settings from the metrics configuration by, say
> {{spark.metrics.props}}. For the backward compatibility, these properties
> would be loaded from the specified as it works now. Such a solution doesn't
> change the API so maybe it could be even included in patch release of Spark
> 1.2 and Spark 1.3.
> Appreciate any feedback.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscr...@spark.apache.org
For additional commands, e-mail: issues-h...@spark.apache.org
