Re: Job can not terminated in Spark 2.0 on Yarn

It works fine now after I remove System.exit(0) 发自我的 iPhone > 在 Aug 2, 2016，12:46 PM，dhruve ashar 写道： > > Can you provide additional logs. > >> On Tue, Aug 2, 2016 at 2:13 PM, Liangzhao Zeng >> wrote: >> It is 2.6 and code is very simple. I load data file fr

Re: Job can not terminated in Spark 2.0 on Yarn

n Tue, Aug 2, 2016 at 10:06 AM, Liangzhao Zeng >> wrote: >> Hi, >> >> I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is >> correct) but get following errors then start over. >> >> Any idea on what happen? >> >>

Re: Job can not terminated in Spark 2.0 on Yarn

ult is > correct) but get following errors then start over." > > Can you elaborate on what do you mean by start over? > > -Dhruve > > > >> On Tue, Aug 2, 2016 at 1:01 PM, Ted Yu wrote: >> Which hadoop version are you using ? >> >> Can

Job can not terminated in Spark 2.0 on Yarn

Hi, I migrate my code to Spark 2.0 from 1.6. It finish last stage (and result is correct) but get following errors then start over. Any idea on what happen? 16/08/02 16:59:33 ERROR scheduler.LiveListenerBus: SparkListenerBus has already stopped! Dropping event SparkListenerExecutorMetricsUpda

How to write Array[Byte] as JPG file in Spark?

Hello All I have RDD[(id:String, image:Array[Byte])] and would like to write the image attribute as a jpg file into HDFS. Any suggestions? Cheers, LZ