You could try tuning spark.shuffle.memoryFraction and spark.storage.memoryFraction (both of which have been deprecated in 1.6), but ultimately you need to find out where you are bottlenecked and address that as adjusting memoryFraction will only be a stopgap. both shuffle and storage memoryFractions default to 0.6
On Wed, Jun 15, 2016 at 9:37 PM, Cassa L <lcas...@gmail.com> wrote: > Hi, > I did set --driver-memory 4G. I still run into this issue after 1 hour > of data load. > > I also tried version 1.6 in test environment. I hit this issue much faster > than in 1.5.1 setup. > LCassa > > On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar <gaura...@gmail.com> > wrote: > >> try setting the option --driver-memory 4G >> >> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater <ben.sla...@instaclustr.com> >> wrote: >> >>> A high level shot in the dark but in our testing we found Spark 1.6 a >>> lot more reliable in low memory situations (presumably due to >>> https://issues.apache.org/jira/browse/SPARK-10000). If it’s an option, >>> probably worth a try. >>> >>> Cheers >>> Ben >>> >>> On Wed, 15 Jun 2016 at 08:48 Cassa L <lcas...@gmail.com> wrote: >>> >>>> Hi, >>>> I would appreciate any clue on this. It has become a bottleneck for our >>>> spark job. >>>> >>>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L <lcas...@gmail.com> wrote: >>>> >>>>> Hi, >>>>> >>>>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark >>>>> and writing it into Cassandra after processing it. Spark job starts fine >>>>> and runs all good for some time until I start getting below errors. Once >>>>> these errors come, job start to lag behind and I see that job has >>>>> scheduling and processing delays in streaming UI. >>>>> >>>>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak >>>>> memoryFraction parameters. Nothing works. >>>>> >>>>> >>>>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with >>>>> curMem=565394, maxMem=2778495713 >>>>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored >>>>> as bytes in memory (estimated size 3.9 KB, free 2.6 GB) >>>>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 >>>>> took 2 ms >>>>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory >>>>> threshold of 1024.0 KB for computing block broadcast_69652 in memory. >>>>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache >>>>> broadcast_69652 in memory! (computed 496.0 B so far) >>>>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 >>>>> GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 >>>>> GB. >>>>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to >>>>> disk instead. >>>>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally >>>>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID >>>>> 452316). 2043 bytes result sent to driver >>>>> >>>>> >>>>> Thanks, >>>>> >>>>> L >>>>> >>>>> >>>> -- >>> ———————— >>> Ben Slater >>> Chief Product Officer >>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support >>> +61 437 929 798 >>> >> >> >
