I do see this message: 15/08/10 19:19:12 WARN YarnScheduler: Initial job has not accepted any resources; check your cluster UI to ensure that workers are registered and have sufficient resources
On Mon, Aug 10, 2015 at 4:15 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > > I am using the same exact code: > > > https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java > > Submitting like this: > > yarn: > > /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/spark-submit --class > org.sony.spark.stream.test.JavaRecoverableNetworkWordCount --master > yarn-client --total-executor-cores 3 > ./spark-streaming-test-0.0.1-SNAPSHOT-jar-with-dependencies.jar localhost > 9999 /user/ec2-user/checkpoint/ /user/ec2-user/out > > local: > > /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/spark-submit --class > org.sony.spark.stream.test.JavaRecoverableNetworkWordCount --master > spark://localhost:9966 --total-executor-cores 3 > ./spark-streaming-test-0.0.1-SNAPSHOT-jar-with-dependencies.jar localhost > 9999 /user/ec2-user/checkpoint/ /user/ec2-user/out > > Even though I am running as local I see it being scheduled and managed by > yarn. > > > On Mon, Aug 10, 2015 at 12:56 PM, Tathagata Das <t...@databricks.com> > wrote: > >> Is it receiving any data? If so, then it must be listening. >> Alternatively, to test these theories, you can locally running a spark >> standalone cluster (one node standalone cluster in local machine), and >> submit your app in client mode on that to see whether you are seeing the >> process listening on 9999 or not. >> >> On Mon, Aug 10, 2015 at 12:43 PM, Mohit Anchlia <mohitanch...@gmail.com> >> wrote: >> >>> I've verified all the executors and I don't see a process listening on >>> the port. However, the application seem to show as running in the yarn UI >>> >>> On Mon, Aug 10, 2015 at 11:56 AM, Tathagata Das <t...@databricks.com> >>> wrote: >>> >>>> In yarn-client mode, the driver is on the machine where you ran the >>>> spark-submit. The executors are running in the YARN cluster nodes, and the >>>> socket receiver listening on port 9999 is running in one of the executors. >>>> >>>> On Mon, Aug 10, 2015 at 11:43 AM, Mohit Anchlia <mohitanch...@gmail.com >>>> > wrote: >>>> >>>>> I am running as a yarn-client which probably means that the program >>>>> that submitted the job is where the listening is also occurring? I thought >>>>> that the yarn is only used to negotiate resources in yarn-client master >>>>> mode. >>>>> >>>>> On Mon, Aug 10, 2015 at 11:34 AM, Tathagata Das <t...@databricks.com> >>>>> wrote: >>>>> >>>>>> If you are running on a cluster, the listening is occurring on one of >>>>>> the executors, not in the driver. >>>>>> >>>>>> On Mon, Aug 10, 2015 at 10:29 AM, Mohit Anchlia < >>>>>> mohitanch...@gmail.com> wrote: >>>>>> >>>>>>> I am trying to run this program as a yarn-client. The job seems to >>>>>>> be submitting successfully however I don't see any process listening on >>>>>>> this host on port 9999 >>>>>>> >>>>>>> >>>>>>> https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java >>>>>>> Active Jobs (2)Job IdDescriptionSubmittedDurationStages: >>>>>>> Succeeded/TotalTasks (for all stages): Succeeded/Total1foreachRDD >>>>>>> at JavaRecoverableNetworkWordCount.java:112 >>>>>>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=1>2015/08/10 >>>>>>> 13:27:3651 s0/2 >>>>>>> 0/2 >>>>>>> 0start at JavaRecoverableNetworkWordCount.java:152 >>>>>>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=0>2015/08/10 >>>>>>> 13:27:3551 s0/2 >>>>>>> 0/70 >>>>>>> >>>>>> >>>>>> >>>>> >>>> >>> >> >
