I changed the place of --files and works.
( IT DOESN'T WORK)
spark-submit --conf spark.metrics.conf=metrics.properties --name
"myProject" --master yarn-cluster --class myCompany.spark.MyClass *--files
/opt/myProject/conf/log4j.properties* --jars $SPARK_CLASSPATH
--executor-memory 1024m --num-exe
I'm trying to configure log4j in Spark.
spark-submit --conf spark.metrics.conf=metrics.properties --name
"myProject" --master yarn-cluster --class myCompany.spark.MyClass *--files
/opt/myProject/conf/log4j.properties* --jars $SPARK_CLASSPATH
--executor-memory 1024m --num-executors 5 --executor-c
Found the issue, a conflict between setting Java options in both
spark-defaults.conf and in the spark-submit.
--
Nick
From: Afshartous, Nick
Sent: Friday, December 18, 2015 11:46 AM
To: user@spark.apache.org
Subject: Configuring log4j
Hi,
Am trying
Hi,
Am trying to configure log4j on an AWS EMR 4.2 Spark cluster for a streaming
job set in client mode.
I changed
/etc/spark/conf/log4j.properties
to use a FileAppender. However the INFO logging still goes to console.
Thanks for any suggestions,
--
Nick
>From the console:
ough this is a single-node cluster so that can't be
> the cause of the error (which is in the driver log).
> --
> Nick
>
> From: Jonathan Kelly [jonathaka...@gmail.com]
> Sent: Thursday, November 19, 2015 6:45 PM
> To: Afsharto
Cc: user@spark.apache.org
Subject: Re: Configuring Log4J (Spark 1.5 on EMR 4.1)
This file only exists on the master and not the slave nodes, so you are
probably running into https://issues.apache.org/jira/browse/SPARK-11105, which
has already been fixed in the not-yet-released Spark 1.6.0. EMR will upgrade t
This file only exists on the master and not the slave nodes, so you are
probably running into https://issues.apache.org/jira/browse/SPARK-11105,
which has already been fixed in the not-yet-released Spark 1.6.0. EMR will
upgrade to Spark 1.6.0 once it is released.
~ Jonathan
On Thu, Nov 19, 2015 a
Hi,
On Spark 1.5 on EMR 4.1 the message below appears in stderr in the Yarn UI.
ERROR StatusLogger No log4j2 configuration file found. Using default
configuration: logging only errors to the console.
I do see that there is
/usr/lib/spark/conf/log4j.properties
Can someone please advise o
