Interesting to hear that it works for you. Are you using Yarn 2.2 as well?
No strange log message during startup, and can't see any other log messages
since no executer gets launched. Does not seems to work in yarn-client mode
either, failing with the exception below.
Exception in thread "main" org.apache.spark.SparkException: Yarn
application has already ended! It might have been killed or unable to
launch application master.
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.waitForApplication(YarnClientSchedulerBackend.scala:119)
at
org.apache.spark.scheduler.cluster.YarnClientSchedulerBackend.start(YarnClientSchedulerBackend.scala:59)
at
org.apache.spark.scheduler.TaskSchedulerImpl.start(TaskSchedulerImpl.scala:141)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:370)
at
com.spotify.analytics.AnalyticsSparkContext.<init>(AnalyticsSparkContext.scala:8)
at com.spotify.analytics.DataSampler$.main(DataSampler.scala:42)
at com.spotify.analytics.DataSampler.main(DataSampler.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
at java.lang.reflect.Method.invoke(Method.java:597)
at
org.apache.spark.deploy.SparkSubmit$.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:551)
at
org.apache.spark.deploy.SparkSubmit$.doRunMain$1(SparkSubmit.scala:155)
at
org.apache.spark.deploy.SparkSubmit$.submit(SparkSubmit.scala:178)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:99)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
/Anders
On Thu, Feb 12, 2015 at 1:33 AM, Sandy Ryza <sandy.r...@cloudera.com> wrote:
> Hi Anders,
>
> I just tried this out and was able to successfully acquire executors. Any
> strange log messages or additional color you can provide on your setup?
> Does yarn-client mode work?
>
> -Sandy
>
> On Wed, Feb 11, 2015 at 1:28 PM, Anders Arpteg <arp...@spotify.com> wrote:
>
>> Hi,
>>
>> Compiled the latest master of Spark yesterday (2015-02-10) for Hadoop
>> 2.2 and failed executing jobs in yarn-cluster mode for that build. Works
>> successfully with spark 1.2 (and also master from 2015-01-16), so something
>> has changed since then that prevents the job from receiving any executors
>> on the cluster.
>>
>> Basic symptoms are that the jobs fires up the AM, but after examining the
>> "executors" page in the web ui, only the driver is listed, no executors
>> are ever received, and the driver keep waiting forever. Has anyone seemed
>> similar problems?
>>
>> Thanks for any insights,
>> Anders
>>
>
>
