Can you give me the whole logs?
TD
On Tue, Feb 10, 2015 at 10:48 AM, Jon Gregg <jonrgr...@gmail.com> wrote:
> OK that worked and getting close here ... the job ran successfully for a
> bit and I got output for the first couple buckets before getting a
> "java.lang.Exception: Could not compute split, block input-0-1423593163000
> not found" error.
>
> So I bumped up the memory at the command line from 2 gb to 5 gb, ran it
> again ... this time I got around 8 successful outputs before erroring.
>
> Bumped up the memory from 5 gb to 10 gb ... got around 15 successful
> outputs before erroring.
>
>
> I'm not persisting or caching anything except for the broadcast IP table
> and another broadcast small user agents list used for the same type of
> filtering, and both files are tiny. The Hadoop cluster is nearly empty
> right now and has more than enough available memory to handle this job. I
> am connecting to Kafka as well and so there's a lot of data coming through
> as my index is trying to catch up to the current date, but yarn-client mode
> has several times in the past few weeks been able to catch up to the
> current date and run successfully for days without issue.
>
> My guess is memory isn't being cleared after each bucket? Relevant
> portion of the log below.
>
>
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593134400 on
> phd40010023.na.com:55551 in memory (size: 50.1 MB, free: 10.2 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593135400 on
> phd40010023.na.com:55551 in memory (size: 24.9 MB, free: 10.2 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593136400 on
> phd40010023.na.com:55551 in memory (size: 129.0 MB, free: 10.3 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593137000 on
> phd40010023.na.com:55551 in memory (size: 112.4 MB, free: 10.4 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593137200 on
> phd40010023.na.com:55551 in memory (size: 481.0 B, free: 10.4 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593137800 on
> phd40010023.na.com:55551 in memory (size: 44.6 MB, free: 10.5 GB)
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 754 from persistence list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593138400 on
> phd40010023.na.com:55551 in memory (size: 95.8 MB, free: 10.6 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 754
> 15/02/10 13:34:54 INFO MapPartitionsRDD: Removing RDD 753 from persistence
> list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593138600 on
> phd40010023.na.com:55551 in memory (size: 123.2 MB, free: 10.7 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593138800 on
> phd40010023.na.com:55551 in memory (size: 5.2 KB, free: 10.7 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 753
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 750 from persistence list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593139600 on
> phd40010023.na.com:55551 in memory (size: 106.4 MB, free: 10.8 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593139800 on
> phd40010023.na.com:55551 in memory (size: 107.0 MB, free: 10.9 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593140000 on
> phd40010023.na.com:55551 in memory (size: 59.5 MB, free: 10.9 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 750
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 749 from persistence list
> 15/02/10 13:34:54 INFO DAGScheduler: Missing parents: List(Stage 117,
> Stage 114, Stage 115, Stage 116)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593140200 on
> phd40010023.na.com:55551 in memory (size: 845.0 B, free: 10.9 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593140600 on
> phd40010023.na.com:55551 in memory (size: 19.2 MB, free: 11.0 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593140800 on
> phd40010023.na.com:55551 in memory (size: 492.0 B, free: 11.0 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593141000 on
> phd40010023.na.com:55551 in memory (size: 1018.0 B, free: 11.0 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 749
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593142400 on
> phd40010023.na.com:55551 in memory (size: 48.6 MB, free: 11.0 GB)
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 767 from persistence list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593142600 on
> phd40010023.na.com:55551 in memory (size: 4.9 KB, free: 11.0 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593142800 on
> phd40010023.na.com:55551 in memory (size: 780.0 B, free: 11.0 GB)
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593143800 on
> phd40010023.na.com:55551 in memory (size: 847.0 B, free: 11.0 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 767
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593144400 on
> phd40010023.na.com:55551 in memory (size: 43.7 MB, free: 11.1 GB)
> 15/02/10 13:34:54 INFO MapPartitionsRDD: Removing RDD 766 from persistence
> list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 766
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 763 from persistence list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593144600 on
> phd40010023.na.com:55551 in memory (size: 827.0 B, free: 11.1 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 763
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 762 from persistence list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593144800 on
> phd40010023.na.com:55551 in memory (size: 1509.0 B, free: 11.1 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 762
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 761 from persistence list
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting Stage 114
> (MapPartitionsRDD[836] at combineByKey at ShuffledDStream.scala:42), which
> has no missing parents
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593147400 on
> phd40010023.na.com:55551 in memory (size: 94.2 MB, free: 11.1 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 761
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 760 from persistence list
> 15/02/10 13:34:54 INFO BlockManagerInfo: Removed input-0-1423593147600 on
> phd40010023.na.com:55551 in memory (size: 75.8 MB, free: 11.2 GB)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 760
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 759 from persistence list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 759
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 758 from persistence list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 758
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting 10 missing tasks from
> Stage 114 (MapPartitionsRDD[836] at combineByKey at
> ShuffledDStream.scala:42)
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 757 from persistence list
> 15/02/10 13:34:54 INFO YarnClusterScheduler: Adding task set 114.0 with 10
> tasks
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 757
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 756 from persistence list
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:0 as TID 2417
> on executor 4: phd40010023.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 756
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:0 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:1 as TID 2418
> on executor 8: phd40010007.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting Stage 115
> (MapPartitionsRDD[842] at combineByKey at ShuffledDStream.scala:42), which
> has no missing parents
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:1 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 783 from persistence list
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:2 as TID 2419
> on executor 6: phd40010002.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:2 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:3 as TID 2420
> on executor 7: phd40010003.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 783
> 15/02/10 13:34:54 INFO MapPartitionsRDD: Removing RDD 782 from persistence
> list
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting 10 missing tasks from
> Stage 115 (MapPartitionsRDD[842] at combineByKey at
> ShuffledDStream.scala:42)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 782
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:3 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO YarnClusterScheduler: Adding task set 115.0 with 10
> tasks
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:4 as TID 2421
> on executor 1: phd40010027.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 779 from persistence list
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:4 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:5 as TID 2422
> on executor 4: phd40010023.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 779
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:5 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:6 as TID 2423
> on executor 8: phd40010007.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 778 from persistence list
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:6 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:7 as TID 2424
> on executor 6: phd40010002.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 778
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:7 as 2024
> bytes in 1 ms
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 777 from persistence list
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:8 as TID 2425
> on executor 7: phd40010003.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:8 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 777
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 114.0:9 as TID 2426
> on executor 1: phd40010027.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 114.0:9 as 2024
> bytes in 0 ms
> 15/02/10 13:34:54 INFO FlatMappedRDD: Removing RDD 776 from persistence
> list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 776
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 775 from persistence list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 775
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 774 from persistence list
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:0 as TID 2427
> on executor 4: phd40010023.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:0 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 774
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:1 as TID 2428
> on executor 6: phd40010002.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:1 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO MapPartitionsRDD: Removing RDD 773 from persistence
> list
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:2 as TID 2429
> on executor 7: phd40010003.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:2 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 773
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:3 as TID 2430
> on executor 8: phd40010007.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:3 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO MappedRDD: Removing RDD 770 from persistence list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 770
> 15/02/10 13:34:54 INFO FilteredRDD: Removing RDD 769 from persistence list
> 15/02/10 13:34:54 INFO BlockManager: Removing RDD 769
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting Stage 116
> (MapPartitionsRDD[855] at combineByKey at ShuffledDStream.scala:42), which
> has no missing parents
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting 10 missing tasks from
> Stage 116 (MapPartitionsRDD[855] at combineByKey at
> ShuffledDStream.scala:42)
> 15/02/10 13:34:54 INFO YarnClusterScheduler: Adding task set 116.0 with 10
> tasks
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting Stage 118
> (MapPartitionsRDD[862] at combineByKey at ShuffledDStream.scala:42), which
> has no missing parents
> 15/02/10 13:34:54 INFO DAGScheduler: Submitting 10 missing tasks from
> Stage 118 (MapPartitionsRDD[862] at combineByKey at
> ShuffledDStream.scala:42)
> 15/02/10 13:34:54 INFO YarnClusterScheduler: Adding task set 118.0 with 10
> tasks
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:4 as TID 2431
> on executor 1: phd40010027.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:4 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:5 as TID 2432
> on executor 1: phd40010027.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:5 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:6 as TID 2433
> on executor 4: phd40010023.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:6 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:7 as TID 2434
> on executor 4: phd40010023.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:7 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 INFO TaskSetManager: Starting task 115.0:8 as TID 2435
> on executor 4: phd40010023.na.com (PROCESS_LOCAL)
> 15/02/10 13:34:54 INFO TaskSetManager: Serialized task 115.0:8 as 1910
> bytes in 0 ms
> 15/02/10 13:34:54 WARN TaskSetManager: Lost TID 2421 (task 114.0:4)
> 15/02/10 13:34:54 WARN TaskSetManager: Loss was due to java.lang.Exception
> java.lang.Exception: Could not compute split, block input-0-1423593163000
> not found
> at org.apache.spark.rdd.BlockRDD.compute(BlockRDD.scala:51)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at org.apache.spark.rdd.FilteredRDD.compute(FilteredRDD.scala:34)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at org.apache.spark.rdd.FlatMappedRDD.compute(FlatMappedRDD.scala:33)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:158)
> at
> org.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:99)
> at org.apache.spark.scheduler.Task.run(Task.scala:51)
> at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:187)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> 15/02/10 13:34:54 WARN TaskSetManager: Lost TID 2426 (task 114.0:9)
> 15/02/10 13:34:54 WARN TaskSetManager: Loss was due to java.lang.Exception
> java.lang.Exception: Could not compute split, block input-0-1423593164000
> not found
> at org.apache.spark.rdd.BlockRDD.compute(BlockRDD.scala:51)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
> at org.apache.spark.rdd.RDD.iterator(RDD.scala:229)
> at org.apache.spark.rdd.MappedRDD.compute(MappedRDD.scala:31)
> at org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:262)
>
> On Tue, Feb 10, 2015 at 1:07 PM, Sandy Ryza <sandy.r...@cloudera.com>
> wrote:
>
>> You should be able to replace that second line with
>>
>> val sc = ssc.sparkContext
>>
>> On Tue, Feb 10, 2015 at 10:04 AM, Jon Gregg <jonrgr...@gmail.com> wrote:
>>
>>> They're separate in my code, how can I combine them? Here's what I have:
>>>
>>> val sparkConf = new SparkConf()
>>> val ssc = new StreamingContext(sparkConf, Seconds(bucketSecs))
>>>
>>> val sc = new SparkContext()
>>>
>>> On Tue, Feb 10, 2015 at 1:02 PM, Sandy Ryza <sandy.r...@cloudera.com>
>>> wrote:
>>>
>>>> Is the SparkContext you're using the same one that the StreamingContext
>>>> wraps? If not, I don't think using two is supported.
>>>>
>>>> -Sandy
>>>>
>>>> On Tue, Feb 10, 2015 at 9:58 AM, Jon Gregg <jonrgr...@gmail.com> wrote:
>>>>
>>>>> I'm still getting an error. Here's my code, which works successfully
>>>>> when tested using spark-shell:
>>>>>
>>>>> val badIPs = sc.textFile("/user/sb/badfullIPs.csv").collect
>>>>> val badIpSet = badIPs.toSet
>>>>> val badIPsBC = sc.broadcast(badIpSet)
>>>>>
>>>>>
>>>>> The job looks OK from my end:
>>>>>
>>>>> 15/02/07 18:59:58 INFO Client: Application report from ASM:
>>>>>
>>>>> application identifier: application_1423081782629_3861
>>>>>
>>>>> appId: 3861
>>>>>
>>>>> * clientToAMToken: Token { kind: YARN_CLIENT_TOKEN, service:
>>>>> }*
>>>>>
>>>>> appDiagnostics:
>>>>>
>>>>> appMasterHost: phd40010008.na.com
>>>>>
>>>>> appQueue: root.default
>>>>>
>>>>> appMasterRpcPort: 0
>>>>>
>>>>> appStartTime: 1423353581140
>>>>>
>>>>> * yarnAppState: RUNNING*
>>>>>
>>>>> distributedFinalState: UNDEFINED
>>>>>
>>>>>
>>>>> But the streaming process never actually begins. The full log is
>>>>> below, scroll to the end for the repeated warning "WARN
>>>>> YarnClusterScheduler: Initial job has not accepted any resources; check
>>>>> your cluster UI to ensure that workers are registered and have sufficient
>>>>> memory".
>>>>>
>>>>> I'll note that I have a different Spark Streaming app called "dqd"
>>>>> working successfully for a different job that uses only a StreamingContext
>>>>> and not an additional SparkContext. But this app (called "sbStreamingTv")
>>>>> uses both a SparkContext and a StreamingContext for grabbing a lookup file
>>>>> in HDFS for IP filtering. * The references to line #198 from the log
>>>>> below refers to the "val badIPs =
>>>>> sc.textFile("/user/sb/badfullIPs.csv").collect" line shown above, and it
>>>>> looks like Spark doesn't get beyond that point in the code.*
>>>>>
>>>>> Also, this job ("sbStreamingTv") does work successfully using
>>>>> yarn-client, even with both a SparkContext and StreamingContext. It looks
>>>>> to me that in yarn-cluster mode it's grabbing resources for the
>>>>> StreamingContext but not for the SparkContext.
>>>>>
>>>>> Any ideas?
>>>>>
>>>>> Jon
>>>>>
>>>>>
>>>>> 15/02/10 12:06:16 INFO MemoryStore: MemoryStore started with capacity
>>>>> 1177.8 MB.
>>>>> 15/02/10 12:06:16 INFO ConnectionManager: Bound socket to port 30129
>>>>> with id = ConnectionManagerId(phd40010008.na.com,30129)
>>>>> 15/02/10 12:06:16 INFO BlockManagerMaster: Trying to register
>>>>> BlockManager
>>>>> 15/02/10 12:06:16 INFO BlockManagerInfo: Registering block manager
>>>>> phd40010008.na.com:30129 with 1177.8 MB RAM
>>>>> 15/02/10 12:06:16 INFO BlockManagerMaster: Registered BlockManager
>>>>> 15/02/10 12:06:16 INFO HttpServer: Starting HTTP Server
>>>>> 15/02/10 12:06:16 INFO HttpBroadcast: Broadcast server started at
>>>>> http://10.229.16.108:35183
>>>>> 15/02/10 12:06:16 INFO HttpFileServer: HTTP File server directory is
>>>>> /hdata/12/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/container_1423081782629_7370_01_000001/tmp/spark-b73a964b-4d91-4af3-8246-48da420c1cec
>>>>> 15/02/10 12:06:16 INFO HttpServer: Starting HTTP Server
>>>>> 15/02/10 12:06:16 INFO JettyUtils: Adding filter:
>>>>> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>>>>> 15/02/10 12:06:16 INFO SparkUI: Started SparkUI at
>>>>> http://phd40010008.na.com:25869
>>>>> 15/02/10 12:06:17 INFO EventLoggingListener: Logging events to
>>>>> /user/spark/applicationHistory/com.na.scalaspark.sbstreamingtv-1423587976801
>>>>> 15/02/10 12:06:17 INFO YarnClusterScheduler: Created
>>>>> YarnClusterScheduler
>>>>> 15/02/10 12:06:17 INFO ApplicationMaster$$anon$1: Adding shutdown hook
>>>>> for context org.apache.spark.SparkContext@7f38095d
>>>>> 15/02/10 12:06:17 INFO ApplicationMaster: Registering the
>>>>> ApplicationMaster
>>>>> 15/02/10 12:06:17 INFO ApplicationMaster: Allocating 3 executors.
>>>>> 15/02/10 12:06:17 INFO YarnAllocationHandler: Will Allocate 3 executor
>>>>> containers, each with 2432 memory
>>>>> 15/02/10 12:06:17 INFO YarnAllocationHandler: Container request (host:
>>>>> Any, priority: 1, capability: <memory:2432, vCores:1>
>>>>> 15/02/10 12:06:17 INFO YarnAllocationHandler: Container request (host:
>>>>> Any, priority: 1, capability: <memory:2432, vCores:1>
>>>>> 15/02/10 12:06:17 INFO YarnAllocationHandler: Container request (host:
>>>>> Any, priority: 1, capability: <memory:2432, vCores:1>
>>>>> 15/02/10 12:06:20 INFO YarnClusterScheduler:
>>>>> YarnClusterScheduler.postStartHook done
>>>>> 15/02/10 12:06:20 WARN SparkConf: In Spark 1.0 and later
>>>>> spark.local.dir will be overridden by the value set by the cluster manager
>>>>> (via SPARK_LOCAL_DIRS in mesos/standalone and LOCAL_DIRS in YARN).
>>>>> 15/02/10 12:06:20 INFO SecurityManager: Changing view acls to: jg
>>>>> 15/02/10 12:06:20 INFO SecurityManager: SecurityManager:
>>>>> authentication disabled; ui acls disabled; users with view permissions:
>>>>> Set(jg)
>>>>> 15/02/10 12:06:20 INFO Slf4jLogger: Slf4jLogger started
>>>>> 15/02/10 12:06:20 INFO Remoting: Starting remoting
>>>>> 15/02/10 12:06:20 INFO Remoting: Remoting started; listening on
>>>>> addresses :[akka.tcp://sp...@phd40010008.na.com:43340]
>>>>> 15/02/10 12:06:20 INFO Remoting: Remoting now listens on addresses:
>>>>> [akka.tcp://sp...@phd40010008.na.com:43340]
>>>>> 15/02/10 12:06:20 INFO SparkEnv: Registering MapOutputTracker
>>>>> 15/02/10 12:06:20 INFO SparkEnv: Registering BlockManagerMaster
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/1/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-f6e1
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/10/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-583d
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/11/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-0b66
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/12/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-bc8f
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/2/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-17e4
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/3/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-c01e
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/4/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-915c
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/5/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-38ff
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/6/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-c92f
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/7/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-b67a
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/8/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-46fb
>>>>> 15/02/10 12:06:20 INFO DiskBlockManager: Created local directory at
>>>>> /hdata/9/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/spark-local-20150210120620-9d11
>>>>> 15/02/10 12:06:20 INFO MemoryStore: MemoryStore started with capacity
>>>>> 1177.8 MB.
>>>>> 15/02/10 12:06:20 INFO ConnectionManager: Bound socket to port 55944
>>>>> with id = ConnectionManagerId(phd40010008.na.com,55944)
>>>>> 15/02/10 12:06:20 INFO BlockManagerMaster: Trying to register
>>>>> BlockManager
>>>>> 15/02/10 12:06:20 INFO BlockManagerInfo: Registering block manager
>>>>> phd40010008.na.com:55944 with 1177.8 MB RAM
>>>>> 15/02/10 12:06:20 INFO BlockManagerMaster: Registered BlockManager
>>>>> 15/02/10 12:06:20 INFO HttpFileServer: HTTP File server directory is
>>>>> /hdata/12/yarn/nm/usercache/jg/appcache/application_1423081782629_7370/container_1423081782629_7370_01_000001/tmp/spark-b3daba9d-f743-4738-b6c2-f56e56813edd
>>>>> 15/02/10 12:06:20 INFO HttpServer: Starting HTTP Server
>>>>> 15/02/10 12:06:20 INFO JettyUtils: Adding filter:
>>>>> org.apache.hadoop.yarn.server.webproxy.amfilter.AmIpFilter
>>>>> 15/02/10 12:06:20 INFO SparkUI: Started SparkUI at
>>>>> http://phd40010008.na.com:10612
>>>>> 15/02/10 12:06:20 INFO EventLoggingListener: Logging events to
>>>>> /user/spark/applicationHistory/com.na.scalaspark.sbstreamingtv-1423587980782
>>>>> 15/02/10 12:06:20 INFO YarnClusterScheduler: Created
>>>>> YarnClusterScheduler
>>>>> 15/02/10 12:06:20 INFO YarnClusterScheduler:
>>>>> YarnClusterScheduler.postStartHook done
>>>>> 15/02/10 12:06:21 INFO MemoryStore: ensureFreeSpace(253715) called
>>>>> with curMem=0, maxMem=1235012812
>>>>> 15/02/10 12:06:21 INFO MemoryStore: Block broadcast_0 stored as values
>>>>> to memory (estimated size 247.8 KB, free 1177.6 MB)
>>>>> 15/02/10 12:06:21 INFO FileInputFormat: Total input paths to process :
>>>>> 1
>>>>> 15/02/10 12:06:21 INFO SparkContext: Starting job: collect at
>>>>> sbStreamingTv.scala:198
>>>>> 15/02/10 12:06:21 INFO DAGScheduler: Got job 0 (collect at
>>>>> sbStreamingTv.scala:198) with 2 output partitions (allowLocal=false)
>>>>> 15/02/10 12:06:21 INFO DAGScheduler: Final stage: Stage 0(*collect at
>>>>> sbStreamingTv.scala:198*)
>>>>> 15/02/10 12:06:21 INFO DAGScheduler: Parents of final stage: List()
>>>>> 15/02/10 12:06:21 INFO DAGScheduler: Missing parents: List()
>>>>> 15/02/10 12:06:21 INFO DAGScheduler: Submitting Stage 0 (*MappedRDD[1]
>>>>> at textFile at sbStreamingTv.scala:198*), which has no missing parents
>>>>> 15/02/10 12:06:21 INFO DAGScheduler: Submitting 2 missing tasks from
>>>>> Stage 0 (*MappedRDD[1] at textFile at sbStreamingTv.scala:198*)
>>>>> 15/02/10 12:06:21 INFO YarnClusterScheduler: Adding task set 0.0 with
>>>>> 2 tasks
>>>>> 15/02/10 12:06:21 INFO AMRMClientImpl: Received new token for :
>>>>> phd40010024.na.com:8041
>>>>> 15/02/10 12:06:21 INFO AMRMClientImpl: Received new token for :
>>>>> phd40010002.na.com:8041
>>>>> 15/02/10 12:06:21 INFO AMRMClientImpl: Received new token for :
>>>>> phd40010022.na.com:8041
>>>>> 15/02/10 12:06:21 INFO RackResolver: Resolved phd40010002.na.com to
>>>>> /sdc/c4h5
>>>>> 15/02/10 12:06:21 INFO RackResolver: Resolved phd40010022.na.com to
>>>>> /sdc/c4h5
>>>>> 15/02/10 12:06:21 INFO RackResolver: Resolved phd40010024.na.com to
>>>>> /sdc/c4h1
>>>>> 15/02/10 12:06:21 INFO YarnAllocationHandler: Launching container
>>>>> container_1423081782629_7370_01_000003 for on host phd40010002.na.com
>>>>> 15/02/10 12:06:21 INFO YarnAllocationHandler: Launching
>>>>> ExecutorRunnable. driverUrl: akka.tcp://
>>>>> sp...@phd40010008.na.com:58240/user/CoarseGrainedScheduler,
>>>>> executorHostname: phd40010002.na.com
>>>>> 15/02/10 12:06:21 INFO YarnAllocationHandler: Launching container
>>>>> container_1423081782629_7370_01_000004 for on host phd40010022.na.com
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Starting Executor Container
>>>>> 15/02/10 12:06:21 INFO YarnAllocationHandler: Launching
>>>>> ExecutorRunnable. driverUrl: akka.tcp://
>>>>> sp...@phd40010008.na.com:58240/user/CoarseGrainedScheduler,
>>>>> executorHostname: phd40010022.na.com
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Starting Executor Container
>>>>> 15/02/10 12:06:21 INFO YarnAllocationHandler: Launching container
>>>>> container_1423081782629_7370_01_000002 for on host phd40010024.na.com
>>>>> 15/02/10 12:06:21 INFO YarnAllocationHandler: Launching
>>>>> ExecutorRunnable. driverUrl: akka.tcp://
>>>>> sp...@phd40010008.na.com:58240/user/CoarseGrainedScheduler,
>>>>> executorHostname: phd40010024.na.com
>>>>> 15/02/10 12:06:21 INFO ContainerManagementProtocolProxy:
>>>>> yarn.client.max-nodemanagers-proxies : 500
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Starting Executor Container
>>>>> 15/02/10 12:06:21 INFO ContainerManagementProtocolProxy:
>>>>> yarn.client.max-nodemanagers-proxies : 500
>>>>> 15/02/10 12:06:21 INFO ContainerManagementProtocolProxy:
>>>>> yarn.client.max-nodemanagers-proxies : 500
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Setting up
>>>>> ContainerLaunchContext
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Setting up
>>>>> ContainerLaunchContext
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Setting up
>>>>> ContainerLaunchContext
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Preparing Local resources
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Preparing Local resources
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Preparing Local resources
>>>>> 15/02/10 12:06:21 INFO ApplicationMaster: All executors have launched.
>>>>> 15/02/10 12:06:21 INFO ApplicationMaster: Started progress reporter
>>>>> thread - sleep time : 5000
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Prepared Local resources
>>>>> Map(__spark__.jar -> resource { scheme: "hdfs" host: "nameservice1" port:
>>>>> -1 file:
>>>>> "/user/jg/.sparkStaging/application_1423081782629_7370/spark-assembly-1.0.0-cdh5.1.3-hadoop2.3.0-cdh5.1.3.jar"
>>>>> } size: 93542713 timestamp: 1423587960750 type: FILE visibility: PRIVATE,
>>>>> __app__.jar -> resource { scheme: "hdfs" host: "nameservice1" port: -1
>>>>> file:
>>>>> "/user/jg/.sparkStaging/application_1423081782629_7370/sbStreamingTv-0.0.1-SNAPSHOT-jar-with-dependencies.jar"
>>>>> } size: 95950353 timestamp: 1423587960370 type: FILE visibility: PRIVATE)
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Prepared Local resources
>>>>> Map(__spark__.jar -> resource { scheme: "hdfs" host: "nameservice1" port:
>>>>> -1 file:
>>>>> "/user/jg/.sparkStaging/application_1423081782629_7370/spark-assembly-1.0.0-cdh5.1.3-hadoop2.3.0-cdh5.1.3.jar"
>>>>> } size: 93542713 timestamp: 1423587960750 type: FILE visibility: PRIVATE,
>>>>> __app__.jar -> resource { scheme: "hdfs" host: "nameservice1" port: -1
>>>>> file:
>>>>> "/user/jg/.sparkStaging/application_1423081782629_7370/sbStreamingTv-0.0.1-SNAPSHOT-jar-with-dependencies.jar"
>>>>> } size: 95950353 timestamp: 1423587960370 type: FILE visibility: PRIVATE)
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Prepared Local resources
>>>>> Map(__spark__.jar -> resource { scheme: "hdfs" host: "nameservice1" port:
>>>>> -1 file:
>>>>> "/user/jg/.sparkStaging/application_1423081782629_7370/spark-assembly-1.0.0-cdh5.1.3-hadoop2.3.0-cdh5.1.3.jar"
>>>>> } size: 93542713 timestamp: 1423587960750 type: FILE visibility: PRIVATE,
>>>>> __app__.jar -> resource { scheme: "hdfs" host: "nameservice1" port: -1
>>>>> file:
>>>>> "/user/jg/.sparkStaging/application_1423081782629_7370/sbStreamingTv-0.0.1-SNAPSHOT-jar-with-dependencies.jar"
>>>>> } size: 95950353 timestamp: 1423587960370 type: FILE visibility: PRIVATE)
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Setting up executor with
>>>>> commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill
>>>>> %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,
>>>>> -Dlog4j.configuration=log4j-spark-container.properties,
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://
>>>>> sp...@phd40010008.na.com:58240/user/CoarseGrainedScheduler, 1,
>>>>> phd40010002.na.com, 1, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Setting up executor with
>>>>> commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill
>>>>> %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,
>>>>> -Dlog4j.configuration=log4j-spark-container.properties,
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://
>>>>> sp...@phd40010008.na.com:58240/user/CoarseGrainedScheduler, 3,
>>>>> phd40010024.na.com, 1, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>>>>> 15/02/10 12:06:21 INFO ExecutorRunnable: Setting up executor with
>>>>> commands: List($JAVA_HOME/bin/java, -server, -XX:OnOutOfMemoryError='kill
>>>>> %p', -Xms2048m -Xmx2048m , -Djava.io.tmpdir=$PWD/tmp,
>>>>> -Dlog4j.configuration=log4j-spark-container.properties,
>>>>> org.apache.spark.executor.CoarseGrainedExecutorBackend, akka.tcp://
>>>>> sp...@phd40010008.na.com:58240/user/CoarseGrainedScheduler, 2,
>>>>> phd40010022.na.com, 1, 1>, <LOG_DIR>/stdout, 2>, <LOG_DIR>/stderr)
>>>>> 15/02/10 12:06:21 INFO ContainerManagementProtocolProxy: Opening proxy
>>>>> : phd40010022.na.com:8041
>>>>> 15/02/10 12:06:21 INFO ContainerManagementProtocolProxy: Opening proxy
>>>>> : phd40010024.na.com:8041
>>>>> 15/02/10 12:06:21 INFO ContainerManagementProtocolProxy: Opening proxy
>>>>> : phd40010002.na.com:8041
>>>>> 15/02/10 12:06:26 INFO CoarseGrainedSchedulerBackend: Registered
>>>>> executor: Actor[akka.tcp://
>>>>> sparkexecu...@phd40010022.na.com:29369/user/Executor#43651774] with
>>>>> ID 2
>>>>> 15/02/10 12:06:26 INFO CoarseGrainedSchedulerBackend: Registered
>>>>> executor: Actor[akka.tcp://
>>>>> sparkexecu...@phd40010024.na.com:12969/user/Executor#1711844295] with
>>>>> ID 3
>>>>> 15/02/10 12:06:26 INFO BlockManagerInfo: Registering block manager
>>>>> phd40010022.na.com:14119 with 1178.1 MB RAM
>>>>> 15/02/10 12:06:26 INFO BlockManagerInfo: Registering block manager
>>>>> phd40010024.na.com:53284 with 1178.1 MB RAM
>>>>> 15/02/10 12:06:29 INFO CoarseGrainedSchedulerBackend: Registered
>>>>> executor: Actor[akka.tcp://
>>>>> sparkexecu...@phd40010002.na.com:35547/user/Executor#-1690254909]
>>>>> with ID 1
>>>>> 15/02/10 12:06:29 INFO BlockManagerInfo: Registering block manager
>>>>> phd40010002.na.com:62754 with 1178.1 MB RAM
>>>>> 15/02/10 12:06:36 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:06:51 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:07:06 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:07:21 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:07:36 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:07:51 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:08:06 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:08:21 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:08:36 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:08:51 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:09:06 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:09:21 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:09:36 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:09:51 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:10:06 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:10:21 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:10:36 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>> 15/02/10 12:10:51 WARN YarnClusterScheduler: Initial job has not
>>>>> accepted any resources; check your cluster UI to ensure that workers are
>>>>> registered and have sufficient memory
>>>>>
>>>>> On Fri, Feb 6, 2015 at 3:24 PM, Sandy Ryza <sandy.r...@cloudera.com>
>>>>> wrote:
>>>>>
>>>>>> You can call collect() to pull in the contents of an RDD into the
>>>>>> driver:
>>>>>>
>>>>>> val badIPsLines = badIPs.collect()
>>>>>>
>>>>>> On Fri, Feb 6, 2015 at 12:19 PM, Jon Gregg <jonrgr...@gmail.com>
>>>>>> wrote:
>>>>>>
>>>>>>> OK I tried that, but how do I convert an RDD to a Set that I can
>>>>>>> then broadcast and cache?
>>>>>>>
>>>>>>> val badIPs = sc.textFile("hdfs:///user/jon/"+ "badfullIPs.csv")
>>>>>>> val badIPsLines = badIPs.getLines
>>>>>>> val badIpSet = badIPsLines.toSet
>>>>>>> val badIPsBC = sc.broadcast(badIpSet)
>>>>>>>
>>>>>>> produces the error "value getLines is not a member of
>>>>>>> org.apache.spark.rdd.RDD[String]".
>>>>>>>
>>>>>>> Leaving it as an RDD and then constantly joining I think will be too
>>>>>>> slow for a streaming job.
>>>>>>>
>>>>>>> On Thu, Feb 5, 2015 at 8:06 PM, Sandy Ryza <sandy.r...@cloudera.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Jon,
>>>>>>>>
>>>>>>>> You'll need to put the file on HDFS (or whatever distributed
>>>>>>>> filesystem you're running on) and load it from there.
>>>>>>>>
>>>>>>>> -Sandy
>>>>>>>>
>>>>>>>> On Thu, Feb 5, 2015 at 3:18 PM, YaoPau <jonrgr...@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> I have a file "badFullIPs.csv" of bad IP addresses used for
>>>>>>>>> filtering. In
>>>>>>>>> yarn-client mode, I simply read it off the edge node, transform
>>>>>>>>> it, and then
>>>>>>>>> broadcast it:
>>>>>>>>>
>>>>>>>>> val badIPs = fromFile(edgeDir + "badfullIPs.csv")
>>>>>>>>> val badIPsLines = badIPs.getLines
>>>>>>>>> val badIpSet = badIPsLines.toSet
>>>>>>>>> val badIPsBC = sc.broadcast(badIpSet)
>>>>>>>>> badIPs.close
>>>>>>>>>
>>>>>>>>> How can I accomplish this in yarn-cluster mode?
>>>>>>>>>
>>>>>>>>> Jon
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> View this message in context:
>>>>>>>>> http://apache-spark-user-list.1001560.n3.nabble.com/How-to-broadcast-a-variable-read-from-a-file-in-yarn-cluster-mode-tp21524.html
>>>>>>>>> Sent from the Apache Spark User List mailing list archive at
>>>>>>>>> Nabble.com.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> ---------------------------------------------------------------------
>>>>>>>>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>>>>>>>>> For additional commands, e-mail: user-h...@spark.apache.org
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>
