Use this in spark conf: spark.ui.showConsoleProgress=false
Best Regards,
On Fri, Apr 24, 2015 at 11:23 AM, Henry Hung wrote:
> Dear All,
>
>
>
> When using spark 1.3.0 spark-submit with directing out and err to a log
> file, I saw some strange lines inside that looks like this:
>
> [Stage 0:>(0 + 2)
> / 120]
>
> [Stage 0:>(2 + 2)
> / 120]
>
> [Stage 0:==> (6 + 2)
> / 120]
>
> [Stage 0:=> (12 + 2)
> / 120]
>
> [Stage 0:=> (20 + 2)
> / 120]
>
> [Stage 0:===>(24 + 2)
> / 120]
>
> [Stage 0:==> (32 + 2)
> / 120]
>
> [Stage 0:===>(42 + 2)
> / 120]
>
> [Stage 0:> (52 + 2)
> / 120]
>
> [Stage 0:===>(59 + 2)
> / 120]
>
> [Stage 0:===>(68 + 2)
> / 120]
>
> [Stage 0:> (78 + 3)
> / 120]
>
> [Stage 0:=> (88 + 4)
> / 120]
>
> [Stage 0:=> (100 + 2)
> / 120]
>
> [Stage 0:==>(110 + 2)
> / 120]
>
>
>
>
>
> Here is my log4j property:
>
>
>
> # Set everything to be logged to the console
>
> log4j.rootCategory=WARN, console
>
> log4j.appender.console=org.apache.log4j.ConsoleAppender
>
> log4j.appender.console.target=System.err
>
> log4j.appender.console.layout=org.apache.log4j.PatternLayout
>
> log4j.appender.console.layout.ConversionPattern=%d{yy/MM/dd HH:mm:ss} %p
> %c{1}: %m%n
>
>
>
> # Settings to quiet third party logs that are too verbose
>
> log4j.logger.org.eclipse.jetty=WARN
>
> log4j.logger.org.eclipse.jetty.util.component.AbstractLifeCycle=ERROR
>
> log4j.logger.org.apache.spark.repl.SparkIMain$exprTyper=INFO
>
> log4j.logger.org.apache.spark.repl.SparkILoop$SparkILoopInterpreter=INFO
>
>
>
>
>
> I want to know how to disable this kind of stage progress message?
>
>
>
> Best regards,
>
> Henry
>
> --
> The privileged confidential information contained in this email is
> intended for use only by the addressees as indicated by the original sender
> of this email. If you are not the addressee indicated in this email or are
> not responsible for delivery of the email to such a person, please kindly
> reply to the sender indicating this fact and delete all copies of it from
> your computer and network server immediately. Your cooperation is highly
> appreciated. It is advised that any unauthorized use of confidential
> information of Winbond is strictly prohibited; and any information in this
> email irrelevant to the official business of Winbond shall be deemed as
> neither given nor endorsed by Winbond.
>
