vidit Singh created SPARK-18987:
-----------------------------------
Summary: I am trying to disable Spark Stage progress logs on
cluster.
Key: SPARK-18987
URL: https://issues.apache.org/jira/browse/SPARK-18987
Project: Spark
Issue Type: Question
Components: Input/Output
Environment: Hadoop Cluster
Reporter: vidit Singh
I am trying to disable Stage progress of spark on Cluster. As i don't want to
see those stages. I have tried every possible things to disable that but still
that problem persist. I have tried the following things :
Logger.getLogger("org").setLevel(Level.OFF);
Logger.getLogger("akka").setLevel(Level.OFF);
Even modified log4j to log4j.logger.org.apache.spark=OFF
I have tried passing spark.ui.showConsoleProgress=false in Spark Submit query.
even passing external log4j along with spark submit didn't work for me. :
--files log4j.properties .
Output :
[Stage 8:> (0 + 2) / 2][Stage 9:> (0 + 2) / 2][Stage 11:> (0 + 2) / 2]
[Stage 8:> (0 + 2) / 2][Stage 9:> (0 + 2) / 2][Stage 13:> (0 + 2) / 2]
[Stage 8:> (0 + 2) / 2][Stage 13:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2]
[Stage 8:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2][Stage 20:==> (1 + 1) / 2]
[Stage 8:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2]
[Stage 8:=========> (1 + 1) / 2][Stage 15:=========> (1 + 1) / 2]
[Stage 15:=============================> (1 + 1) / 2]
[Stage 10:> (0 + 180) / 200]
[Stage 10:> (0 + 180) / 200][Stage 17:> (0 + 0) / 200]
[Stage 10:> (2 + 180) / 200][Stage 17:> (0 + 0) / 200]
[Stage 10:> (5 + 180) / 200][Stage 17:> (0 + 0) / 200]
[Stage 10:> (14 + 180) / 200][Stage 17:> (0 + 0) / 200]
I don't want to see these stages in my logs. Please help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
