1. When you are running locally, make sure the "master" in the SparkConf reflects that and is not somehow set to "yarn-client"
2. You may not be getting any resources from YARN at all, so no executors, so no receiver running. That is why I asked the most basic question - Is it receiving data? That will eliminate a lot of uncertainities if it is indeed receiving data. TD On Mon, Aug 10, 2015 at 4:21 PM, Mohit Anchlia <mohitanch...@gmail.com> wrote: > I do see this message: > > 15/08/10 19:19:12 WARN YarnScheduler: Initial job has not accepted any > resources; check your cluster UI to ensure that workers are registered and > have sufficient resources > > On Mon, Aug 10, 2015 at 4:15 PM, Mohit Anchlia <mohitanch...@gmail.com> > wrote: > >> >> I am using the same exact code: >> >> >> https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java >> >> Submitting like this: >> >> yarn: >> >> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/spark-submit --class >> org.sony.spark.stream.test.JavaRecoverableNetworkWordCount --master >> yarn-client --total-executor-cores 3 >> ./spark-streaming-test-0.0.1-SNAPSHOT-jar-with-dependencies.jar localhost >> 9999 /user/ec2-user/checkpoint/ /user/ec2-user/out >> >> local: >> >> /opt/cloudera/parcels/CDH-5.4.0-1.cdh5.4.0.p0.27/bin/spark-submit --class >> org.sony.spark.stream.test.JavaRecoverableNetworkWordCount --master >> spark://localhost:9966 --total-executor-cores 3 >> ./spark-streaming-test-0.0.1-SNAPSHOT-jar-with-dependencies.jar localhost >> 9999 /user/ec2-user/checkpoint/ /user/ec2-user/out >> >> Even though I am running as local I see it being scheduled and managed by >> yarn. >> >> >> On Mon, Aug 10, 2015 at 12:56 PM, Tathagata Das <t...@databricks.com> >> wrote: >> >>> Is it receiving any data? If so, then it must be listening. >>> Alternatively, to test these theories, you can locally running a spark >>> standalone cluster (one node standalone cluster in local machine), and >>> submit your app in client mode on that to see whether you are seeing the >>> process listening on 9999 or not. >>> >>> On Mon, Aug 10, 2015 at 12:43 PM, Mohit Anchlia <mohitanch...@gmail.com> >>> wrote: >>> >>>> I've verified all the executors and I don't see a process listening on >>>> the port. However, the application seem to show as running in the yarn UI >>>> >>>> On Mon, Aug 10, 2015 at 11:56 AM, Tathagata Das <t...@databricks.com> >>>> wrote: >>>> >>>>> In yarn-client mode, the driver is on the machine where you ran the >>>>> spark-submit. The executors are running in the YARN cluster nodes, and the >>>>> socket receiver listening on port 9999 is running in one of the executors. >>>>> >>>>> On Mon, Aug 10, 2015 at 11:43 AM, Mohit Anchlia < >>>>> mohitanch...@gmail.com> wrote: >>>>> >>>>>> I am running as a yarn-client which probably means that the program >>>>>> that submitted the job is where the listening is also occurring? I >>>>>> thought >>>>>> that the yarn is only used to negotiate resources in yarn-client master >>>>>> mode. >>>>>> >>>>>> On Mon, Aug 10, 2015 at 11:34 AM, Tathagata Das <t...@databricks.com> >>>>>> wrote: >>>>>> >>>>>>> If you are running on a cluster, the listening is occurring on one >>>>>>> of the executors, not in the driver. >>>>>>> >>>>>>> On Mon, Aug 10, 2015 at 10:29 AM, Mohit Anchlia < >>>>>>> mohitanch...@gmail.com> wrote: >>>>>>> >>>>>>>> I am trying to run this program as a yarn-client. The job seems to >>>>>>>> be submitting successfully however I don't see any process listening on >>>>>>>> this host on port 9999 >>>>>>>> >>>>>>>> >>>>>>>> https://github.com/apache/spark/blob/master/examples/src/main/java/org/apache/spark/examples/streaming/JavaRecoverableNetworkWordCount.java >>>>>>>> Active Jobs (2)Job IdDescriptionSubmittedDurationStages: >>>>>>>> Succeeded/TotalTasks (for all stages): Succeeded/Total1foreachRDD >>>>>>>> at JavaRecoverableNetworkWordCount.java:112 >>>>>>>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=1>2015/08/10 >>>>>>>> 13:27:3651 s0/2 >>>>>>>> 0/2 >>>>>>>> 0start at JavaRecoverableNetworkWordCount.java:152 >>>>>>>> <http://ec2-52-25-118-171.us-west-2.compute.amazonaws.com:8088/proxy/application_1438820875993_0007/jobs/job?id=0>2015/08/10 >>>>>>>> 13:27:3551 s0/2 >>>>>>>> 0/70 >>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>> >>> >> >
