[
https://issues.apache.org/jira/browse/SPARK-18987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sean Owen resolved SPARK-18987.
-------------------------------
Resolution: Invalid
Target Version/s: (was: 1.5.1)
Please read http://spark.apache.org/contributing.html
Questions go to [email protected]
> I am trying to disable Spark Stage progress logs on cluster.
> ------------------------------------------------------------
>
> Key: SPARK-18987
> URL: https://issues.apache.org/jira/browse/SPARK-18987
> Project: Spark
> Issue Type: Question
> Components: Input/Output
> Environment: Hadoop Cluster
> Reporter: vidit Singh
>
> I am trying to disable Stage progress of spark on Cluster. As i don't want
> to see those stages. I have tried every possible things to disable that but
> still that problem persist. I have tried the following things :
> Logger.getLogger("org").setLevel(Level.OFF);
> Logger.getLogger("akka").setLevel(Level.OFF);
> Even modified log4j to log4j.logger.org.apache.spark=OFF
> I have tried passing spark.ui.showConsoleProgress=false in Spark Submit
> query. even passing external log4j along with spark submit didn't work for
> me. : --files log4j.properties .
> Output :
> [Stage 8:> (0 + 2) / 2][Stage 9:> (0 + 2) / 2][Stage 11:> (0 + 2) / 2]
> [Stage 8:> (0 + 2) / 2][Stage 9:> (0 + 2) / 2][Stage 13:> (0 + 2) / 2]
> [Stage 8:> (0 + 2) / 2][Stage 13:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2]
> [Stage 8:> (0 + 2) / 2][Stage 15:> (0 + 2) / 2][Stage 20:==> (1 + 1) / 2]
> [Stage 8:> (0 + 2) / 2][Stage 15:> (0 + 2) /
> 2]
> [Stage 8:=========> (1 + 1) / 2][Stage 15:=========> (1 + 1) /
> 2]
> [Stage 15:=============================> (1 + 1) /
> 2]
> [Stage 10:> (0 + 180) /
> 200]
> [Stage 10:> (0 + 180) / 200][Stage 17:> (0 + 0) /
> 200]
> [Stage 10:> (2 + 180) / 200][Stage 17:> (0 + 0) /
> 200]
> [Stage 10:> (5 + 180) / 200][Stage 17:> (0 + 0) /
> 200]
> [Stage 10:> (14 + 180) / 200][Stage 17:> (0 + 0) /
> 200]
> I don't want to see these stages in my logs. Please help.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
