Use this in spark conf: spark.ui.showConsoleProgress=false Best Regards,
On Fri, Apr 24, 2015 at 11:23 AM, Henry Hung <ythu...@winbond.com> wrote: > Dear All, > > > > When using spark 1.3.0 spark-submit with directing out and err to a log > file, I saw some strange lines inside that looks like this: > > [Stage 0:> (0 + 2) > / 120] > > [Stage 0:> (2 + 2) > / 120] > > [Stage 0:==> (6 + 2) > / 120] > > [Stage 0:=====> (12 + 2) > / 120] > > [Stage 0:=========> (20 + 2) > / 120] > > [Stage 0:===========> (24 + 2) > / 120] > > [Stage 0:==============> (32 + 2) > / 120] > > [Stage 0:===================> (42 + 2) > / 120] > > [Stage 0:========================> (52 + 2) > / 120] > > [Stage 0:===========================> (59 + 2) > / 120] > > [Stage 0:===============================> (68 + 2) > / 120] > > [Stage 0:====================================> (78 + 3) > / 120] > > [Stage 0:=========================================> (88 + 4) > / 120] > > [Stage 0:=============================================> (100 + 2) > / 120] > > [Stage 0:==================================================> (110 + 2) > / 120] > > > > > > Here is my log4j property: > > > > # Set everything to be logged to the console > > log4j.rootCategory=WARN, console > > log4j.appender.console=org.apache.log4j.ConsoleAppender > > log4j.appender.console.target=System.err > > log4j.appender.console.layout=org.apache.log4j.PatternLayout > > log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p > %c{1}: %m%n > > > > # Settings to quiet third party logs that are too verbose > > log4j.logger.org.eclipse.jetty=WARN > > log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR > > log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO > > log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO > > > > > > I want to know how to disable this kind of stage progress message? > > > > Best regards, > > Henry > > ------------------------------ > The privileged confidential information contained in this email is > intended for use only by the addressees as indicated by the original sender > of this email. If you are not the addressee indicated in this email or are > not responsible for delivery of the email to such a person, please kindly > reply to the sender indicating this fact and delete all copies of it from > your computer and network server immediately. Your cooperation is highly > appreciated. It is advised that any unauthorized use of confidential > information of Winbond is strictly prohibited; and any information in this > email irrelevant to the official business of Winbond shall be deemed as > neither given nor endorsed by Winbond. >