Re: EMR Spark log4j and metrics

I wonder if anyone can confirm is Spark on YARN the problem here? Or is it how AWS has put it together? I'm wondering if Spark on YARN has problems with configuration files for the workers and driver? Peter Halliday On Thu, Apr 14, 2016 at 1:09 PM, Peter Halliday wrote:

Re: EMR Spark log4j and metrics

An update to this is that I can see the log4j.properties files and the metrics.properties files correctly on the master. When I submit a Spark Step that runs Spark in deploy mode of cluster, I see the cluster files being zipped up and pushed via hdfs to the driver and workers. However, I don't

EMR Spark log4j and metrics

I have an existing cluster that I stand up via Docker images and CloudFormation Templates on AWS. We are moving to EMR and AWS Data Pipeline process, and having problems with metrics and log4j. We’ve sent a JSON configuration for spark-log4j and spark-metrics. The log4j file seems to be