Re: Accessing SparkConf in metrics sink
So the answer to my previous question is NO. It looks like I could use SparkEnv.get.conf but * * NOTE: This is not intended for external use. This is exposed for Shark and may be made private * in a future release. */ On Wed, 16 Mar 2016 at 08:22 Pete Robbins wrote: > OK thanks. Does that work in an executor? > > On Wed, 16 Mar 2016 at 07:58 Reynold Xin wrote: > >> SparkConf is not a singleton. >> >> However, SparkContext in almost all cases are. So you can use >> SparkContext.getOrCreate().getConf >> >> On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins >> wrote: >> >>> I'm writing a metrics sink and reporter to push metrics to >>> Elasticsearch. An example format of a metric in JSON: >>> >>> { >>> "timestamp": "2016-03-15T16:11:19.314+", >>> "hostName": "10.192.0.87" >>> "applicationName": "My application", >>> "applicationId": "app-20160315093931-0003", >>> "executorId": "17", >>> "executor_threadpool_completeTasks": 20 >>> } >>> >>> For correlating the metrics I want the timestamp, hostname, >>> applicationId, executorId and applicationName. >>> >>> Currently I am extracting the applicationId and executor Id from the >>> metric name as MetricsSystem prepends these to the name. As the sink is >>> instantiated without the SparkConf I can not determine the applicationName. >>> >>> Another proposed change in >>> https://issues.apache.org/jira/browse/SPARK-10610 would also make me >>> require access to the SparkConf to get the applicationId/executorId. >>> >>> So, Is the SparkConf a singleton and can there be a Utils method for >>> accessing it? Instantiating a SparkConf myself will not pick up the appName >>> etc as these are set via methods on the conf. >>> >>> I'm trying to write this without modifying any Spark code by just using >>> a definition in the metrics properties to load my sink. >>> >>> Cheers, >>> >> >>
Re: Accessing SparkConf in metrics sink
OK thanks. Does that work in an executor? On Wed, 16 Mar 2016 at 07:58 Reynold Xin wrote: > SparkConf is not a singleton. > > However, SparkContext in almost all cases are. So you can use > SparkContext.getOrCreate().getConf > > On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins > wrote: > >> I'm writing a metrics sink and reporter to push metrics to Elasticsearch. >> An example format of a metric in JSON: >> >> { >> "timestamp": "2016-03-15T16:11:19.314+", >> "hostName": "10.192.0.87" >> "applicationName": "My application", >> "applicationId": "app-20160315093931-0003", >> "executorId": "17", >> "executor_threadpool_completeTasks": 20 >> } >> >> For correlating the metrics I want the timestamp, hostname, >> applicationId, executorId and applicationName. >> >> Currently I am extracting the applicationId and executor Id from the >> metric name as MetricsSystem prepends these to the name. As the sink is >> instantiated without the SparkConf I can not determine the applicationName. >> >> Another proposed change in >> https://issues.apache.org/jira/browse/SPARK-10610 would also make me >> require access to the SparkConf to get the applicationId/executorId. >> >> So, Is the SparkConf a singleton and can there be a Utils method for >> accessing it? Instantiating a SparkConf myself will not pick up the appName >> etc as these are set via methods on the conf. >> >> I'm trying to write this without modifying any Spark code by just using a >> definition in the metrics properties to load my sink. >> >> Cheers, >> > >
Re: Accessing SparkConf in metrics sink
SparkConf is not a singleton. However, SparkContext in almost all cases are. So you can use SparkContext.getOrCreate().getConf On Wed, Mar 16, 2016 at 12:38 AM, Pete Robbins wrote: > I'm writing a metrics sink and reporter to push metrics to Elasticsearch. > An example format of a metric in JSON: > > { > "timestamp": "2016-03-15T16:11:19.314+", > "hostName": "10.192.0.87" > "applicationName": "My application", > "applicationId": "app-20160315093931-0003", > "executorId": "17", > "executor_threadpool_completeTasks": 20 > } > > For correlating the metrics I want the timestamp, hostname, applicationId, > executorId and applicationName. > > Currently I am extracting the applicationId and executor Id from the > metric name as MetricsSystem prepends these to the name. As the sink is > instantiated without the SparkConf I can not determine the applicationName. > > Another proposed change in > https://issues.apache.org/jira/browse/SPARK-10610 would also make me > require access to the SparkConf to get the applicationId/executorId. > > So, Is the SparkConf a singleton and can there be a Utils method for > accessing it? Instantiating a SparkConf myself will not pick up the appName > etc as these are set via methods on the conf. > > I'm trying to write this without modifying any Spark code by just using a > definition in the metrics properties to load my sink. > > Cheers, >
Accessing SparkConf in metrics sink
I'm writing a metrics sink and reporter to push metrics to Elasticsearch. An example format of a metric in JSON: { "timestamp": "2016-03-15T16:11:19.314+", "hostName": "10.192.0.87" "applicationName": "My application", "applicationId": "app-20160315093931-0003", "executorId": "17", "executor_threadpool_completeTasks": 20 } For correlating the metrics I want the timestamp, hostname, applicationId, executorId and applicationName. Currently I am extracting the applicationId and executor Id from the metric name as MetricsSystem prepends these to the name. As the sink is instantiated without the SparkConf I can not determine the applicationName. Another proposed change in https://issues.apache.org/jira/browse/SPARK-10610 would also make me require access to the SparkConf to get the applicationId/executorId. So, Is the SparkConf a singleton and can there be a Utils method for accessing it? Instantiating a SparkConf myself will not pick up the appName etc as these are set via methods on the conf. I'm trying to write this without modifying any Spark code by just using a definition in the metrics properties to load my sink. Cheers,