Re: Spark Memory Error - Not enough space to cache broadcast

[Dennis Lovely]

I believe you want to set memoryFraction higher, not lower.  These two
older threads seem to have similar issues you are experiencing:

https://mail-archives.apache.org/mod_mbox/spark-user/201503.mbox/%3CCAHUQ+_ZqaWFs_MJ=+V49bD2paKvjLErPKMEW5duLO1jAo4=d...@mail.gmail.com%3E
https://www.mail-archive.com/user@spark.apache.org/msg44793.html

More info on tuning shuffle behavior:
https://spark.apache.org/docs/1.5.1/configuration.html#shuffle-behavior

On Thu, Jun 16, 2016 at 1:57 PM, Cassa L  wrote:

> Hi Dennis,
>
> On Wed, Jun 15, 2016 at 11:39 PM, Dennis Lovely  wrote:
>
>> You could try tuning spark.shuffle.memoryFraction and
>> spark.storage.memoryFraction (both of which have been deprecated in 1.6),
>> but ultimately you need to find out where you are bottlenecked and address
>> that as adjusting memoryFraction will only be a stopgap.  both shuffle and
>> storage memoryFractions default to 0.6
>>
>> I have set above parameters to 0.5. Does it need to increased?
>
> Thanks.
>
>> On Wed, Jun 15, 2016 at 9:37 PM, Cassa L  wrote:
>>
>>> Hi,
>>>  I did set  --driver-memory 4G. I still run into this issue after 1
>>> hour of data load.
>>>
>>> I also tried version 1.6 in test environment. I hit this issue much
>>> faster than in 1.5.1 setup.
>>> LCassa
>>>
>>> On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
>>> wrote:
>>>
 try setting the option --driver-memory 4G

 On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater  wrote:

> A high level shot in the dark but in our testing we found Spark 1.6 a
> lot more reliable in low memory situations (presumably due to
> https://issues.apache.org/jira/browse/SPARK-1). If it’s an
> option, probably worth a try.
>
> Cheers
> Ben
>
> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>
>> Hi,
>> I would appreciate any clue on this. It has become a bottleneck for
>> our spark job.
>>
>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>>
>>> Hi,
>>>
>>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark 
>>> and writing it into Cassandra after processing it. Spark job starts 
>>> fine and runs all good for some time until I start getting below 
>>> errors. Once these errors come, job start to lag behind and I see that 
>>> job has scheduling and processing delays in streaming  UI.
>>>
>>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>>> memoryFraction parameters. Nothing works.
>>>
>>>
>>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>>> curMem=565394, maxMem=2778495713
>>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored 
>>> as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 
>>> 69652 took 2 ms
>>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>>> broadcast_69652 in memory! (computed 496.0 B so far)
>>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 
>>> 2.6 GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit 
>>> = 2.6 GB.
>>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
>>> disk instead.
>>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 
>>> (TID 452316). 2043 bytes result sent to driver
>>>
>>>
>>> Thanks,
>>>
>>> L
>>>
>>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>


>>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi Dennis,

On Wed, Jun 15, 2016 at 11:39 PM, Dennis Lovely  wrote:

> You could try tuning spark.shuffle.memoryFraction and
> spark.storage.memoryFraction (both of which have been deprecated in 1.6),
> but ultimately you need to find out where you are bottlenecked and address
> that as adjusting memoryFraction will only be a stopgap.  both shuffle and
> storage memoryFractions default to 0.6
>
> I have set above parameters to 0.5. Does it need to increased?

Thanks.

> On Wed, Jun 15, 2016 at 9:37 PM, Cassa L  wrote:
>
>> Hi,
>>  I did set  --driver-memory 4G. I still run into this issue after 1 hour
>> of data load.
>>
>> I also tried version 1.6 in test environment. I hit this issue much
>> faster than in 1.5.1 setup.
>> LCassa
>>
>> On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
>> wrote:
>>
>>> try setting the option --driver-memory 4G
>>>
>>> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
>>> wrote:
>>>
 A high level shot in the dark but in our testing we found Spark 1.6 a
 lot more reliable in low memory situations (presumably due to
 https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
 probably worth a try.

 Cheers
 Ben

 On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:

> Hi,
> I would appreciate any clue on this. It has become a bottleneck for
> our spark job.
>
> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>
>> Hi,
>>
>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark 
>> and writing it into Cassandra after processing it. Spark job starts fine 
>> and runs all good for some time until I start getting below errors. Once 
>> these errors come, job start to lag behind and I see that job has 
>> scheduling and processing delays in streaming  UI.
>>
>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>> memoryFraction parameters. Nothing works.
>>
>>
>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>> curMem=565394, maxMem=2778495713
>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored 
>> as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 
>> 69652 took 2 ms
>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>> broadcast_69652 in memory! (computed 496.0 B so far)
>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 
>> GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 
>> 2.6 GB.
>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
>> disk instead.
>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
>> 452316). 2043 bytes result sent to driver
>>
>>
>> Thanks,
>>
>> L
>>
>>
> --
 
 Ben Slater
 Chief Product Officer
 Instaclustr: Cassandra + Spark - Managed | Consulting | Support
 +61 437 929 798

>>>
>>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

On Thu, Jun 16, 2016 at 5:27 AM, Deepak Goel  wrote:

> What is your hardware configuration like which you are running Spark on?
>
> It  is 24core, 128GB RAM

> Hey
>
> Namaskara~Nalama~Guten Tag~Bonjour
>
>
>--
> Keigu
>
> Deepak
> 73500 12833
> www.simtree.net, dee...@simtree.net
> deic...@gmail.com
>
> LinkedIn: www.linkedin.com/in/deicool
> Skype: thumsupdeicool
> Google talk: deicool
> Blog: http://loveandfearless.wordpress.com
> Facebook: http://www.facebook.com/deicool
>
> "Contribute to the world, environment and more :
> http://www.gridrepublic.org
> "
>
> On Thu, Jun 16, 2016 at 5:33 PM, Jacek Laskowski  wrote:
>
>> Hi,
>>
>> What do you see under Executors and Details for Stage (for the
>> affected stages)? Anything weird memory-related?
>>
>> How does your "I am reading data from Kafka into Spark and writing it
>> into Cassandra after processing it." pipeline look like?
>>
>> Pozdrawiam,
>> Jacek Laskowski
>> 
>> https://medium.com/@jaceklaskowski/
>> Mastering Apache Spark http://bit.ly/mastering-apache-spark
>> Follow me at https://twitter.com/jaceklaskowski
>>
>>
>> On Mon, Jun 13, 2016 at 11:56 PM, Cassa L  wrote:
>> > Hi,
>> >
>> > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark
>> and
>> > writing it into Cassandra after processing it. Spark job starts fine and
>> > runs all good for some time until I start getting below errors. Once
>> these
>> > errors come, job start to lag behind and I see that job has scheduling
>> and
>> > processing delays in streaming  UI.
>> >
>> > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
>> > memoryFraction parameters. Nothing works.
>> >
>> >
>> > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
>> > curMem=565394, maxMem=2778495713
>> > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored
>> as
>> > bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>> > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable
>> 69652
>> > took 2 ms
>> > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
>> > threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>> > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
>> > broadcast_69652 in memory! (computed 496.0 B so far)
>> > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) +
>> 2.6 GB
>> > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6
>> GB.
>> > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to
>> disk
>> > instead.
>> > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>> > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID
>> > 452316). 2043 bytes result sent to driver
>> >
>> >
>> > Thanks,
>> >
>> > L
>>
>> -
>> To unsubscribe, e-mail: user-unsubscr...@spark.apache.org
>> For additional commands, e-mail: user-h...@spark.apache.org
>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,

>
> What do you see under Executors and Details for Stage (for the
> affected stages)? Anything weird memory-related?
>
Under executor Tab, logs throw these warning -

16/06/16 20:45:40 INFO TorrentBroadcast: Reading broadcast variable
422145 took 1 ms
16/06/16 20:45:40 WARN MemoryStore: Failed to reserve initial memory
threshold of 1024.0 KB for computing block broadcast_422145 in memory.
16/06/16 20:45:40 WARN MemoryStore: Not enough space to cache
broadcast_422145 in memory! (computed 496.0 B so far)
16/06/16 20:45:40 INFO MemoryStore: Memory use = 147.9 KB (blocks) +
2.2 GB (scratch space shared across 0 tasks(s)) = 2.2 GB. Storage
limit = 2.2 GB.
16/06/16 20:45:40 WARN MemoryStore: Persisting block broadcast_422145
to disk instead.
16/06/16 20:45:40 INFO MapOutputTrackerWorker: Don't have map outputs
for shuffle 70278, fetching them

16/06/16 20:45:40 INFO MapOutputTrackerWorker: Doing the fetch; tracker
endpoint = AkkaRpcEndpointRef(Actor[akka.tcp://
sparkDriver@17.40.240.71:46187/user/MapOutputTracker#-1794035569])

I dont see any memory related errors on 'stages' Tab.

>
> How does your "I am reading data from Kafka into Spark and writing it
> into Cassandra after processing it." pipeline look like?
>
> This part has no issues. Reading from Kafka is always up to date. There
are no offset lags. Writting to Cassandra is also fine with less than 1ms
to write data.


> Pozdrawiam,
> Jacek Laskowski
> 
> https://medium.com/@jaceklaskowski/
> Mastering Apache Spark http://bit.ly/mastering-apache-spark
> Follow me at https://twitter.com/jaceklaskowski
>
>
> On Mon, Jun 13, 2016 at 11:56 PM, Cassa L  wrote:
> > Hi,
> >
> > I'm using spark 1.5.1 version. I am reading data from Kafka into Spark
> and
> > writing it into Cassandra after processing it. Spark job starts fine and
> > runs all good for some time until I start getting below errors. Once
> these
> > errors come, job start to lag behind and I see that job has scheduling
> and
> > processing delays in streaming  UI.
> >
> > Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
> > memoryFraction parameters. Nothing works.
> >
> >
> > 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
> > curMem=565394, maxMem=2778495713
> > 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored
> as
> > bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> > 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652
> > took 2 ms
> > 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
> > threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> > 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
> > broadcast_69652 in memory! (computed 496.0 B so far)
> > 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6
> GB
> > (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6
> GB.
> > 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to
> disk
> > instead.
> > 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> > 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID
> > 452316). 2043 bytes result sent to driver
> >
> >
> > Thanks,
> >
> > L
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Jacek Laskowski]

Hi,

What do you see under Executors and Details for Stage (for the
affected stages)? Anything weird memory-related?

How does your "I am reading data from Kafka into Spark and writing it
into Cassandra after processing it." pipeline look like?

Pozdrawiam,
Jacek Laskowski

https://medium.com/@jaceklaskowski/
Mastering Apache Spark http://bit.ly/mastering-apache-spark
Follow me at https://twitter.com/jaceklaskowski


On Mon, Jun 13, 2016 at 11:56 PM, Cassa L  wrote:
> Hi,
>
> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and
> writing it into Cassandra after processing it. Spark job starts fine and
> runs all good for some time until I start getting below errors. Once these
> errors come, job start to lag behind and I see that job has scheduling and
> processing delays in streaming  UI.
>
> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak
> memoryFraction parameters. Nothing works.
>
>
> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with
> curMem=565394, maxMem=2778495713
> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as
> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652
> took 2 ms
> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory
> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache
> broadcast_69652 in memory! (computed 496.0 B so far)
> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB
> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to disk
> instead.
> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID
> 452316). 2043 bytes result sent to driver
>
>
> Thanks,
>
> L

Re: Spark Memory Error - Not enough space to cache broadcast

[Dennis Lovely]

You could try tuning spark.shuffle.memoryFraction and
spark.storage.memoryFraction (both of which have been deprecated in 1.6),
but ultimately you need to find out where you are bottlenecked and address
that as adjusting memoryFraction will only be a stopgap.  both shuffle and
storage memoryFractions default to 0.6

On Wed, Jun 15, 2016 at 9:37 PM, Cassa L  wrote:

> Hi,
>  I did set  --driver-memory 4G. I still run into this issue after 1 hour
> of data load.
>
> I also tried version 1.6 in test environment. I hit this issue much faster
> than in 1.5.1 setup.
> LCassa
>
> On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
> wrote:
>
>> try setting the option --driver-memory 4G
>>
>> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
>> wrote:
>>
>>> A high level shot in the dark but in our testing we found Spark 1.6 a
>>> lot more reliable in low memory situations (presumably due to
>>> https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
>>> probably worth a try.
>>>
>>> Cheers
>>> Ben
>>>
>>> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>>>
 Hi,
 I would appreciate any clue on this. It has become a bottleneck for our
 spark job.

 On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:

> Hi,
>
> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark 
> and writing it into Cassandra after processing it. Spark job starts fine 
> and runs all good for some time until I start getting below errors. Once 
> these errors come, job start to lag behind and I see that job has 
> scheduling and processing delays in streaming  UI.
>
> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
> memoryFraction parameters. Nothing works.
>
>
> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
> curMem=565394, maxMem=2778495713
> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored 
> as bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
> took 2 ms
> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
> broadcast_69652 in memory! (computed 496.0 B so far)
> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 
> GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 
> GB.
> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
> disk instead.
> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
> 452316). 2043 bytes result sent to driver
>
>
> Thanks,
>
> L
>
>
 --
>>> 
>>> Ben Slater
>>> Chief Product Officer
>>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>>> +61 437 929 798
>>>
>>
>>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,
 I did set  --driver-memory 4G. I still run into this issue after 1 hour of
data load.

I also tried version 1.6 in test environment. I hit this issue much faster
than in 1.5.1 setup.
LCassa

On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
wrote:

> try setting the option --driver-memory 4G
>
> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
> wrote:
>
>> A high level shot in the dark but in our testing we found Spark 1.6 a lot
>> more reliable in low memory situations (presumably due to
>> https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
>> probably worth a try.
>>
>> Cheers
>> Ben
>>
>> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>>
>>> Hi,
>>> I would appreciate any clue on this. It has become a bottleneck for our
>>> spark job.
>>>
>>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>>>
 Hi,

 I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
 writing it into Cassandra after processing it. Spark job starts fine and 
 runs all good for some time until I start getting below errors. Once these 
 errors come, job start to lag behind and I see that job has scheduling and 
 processing delays in streaming  UI.

 Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
 memoryFraction parameters. Nothing works.


 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
 curMem=565394, maxMem=2778495713
 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
 bytes in memory (estimated size 3.9 KB, free 2.6 GB)
 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
 took 2 ms
 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
 threshold of 1024.0 KB for computing block broadcast_69652 in memory.
 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
 broadcast_69652 in memory! (computed 496.0 B so far)
 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 
 GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 
 GB.
 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
 disk instead.
 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
 452316). 2043 bytes result sent to driver


 Thanks,

 L


>>> --
>> 
>> Ben Slater
>> Chief Product Officer
>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>> +61 437 929 798
>>
>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,
Upgrading sprak is not option right now. I did set  --driver-memory 4G. I
still run into this issue after 1 hour of data load.

LCassa


On Tue, Jun 14, 2016 at 3:57 PM, Gaurav Bhatnagar 
wrote:

> try setting the option --driver-memory 4G
>
> On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
> wrote:
>
>> A high level shot in the dark but in our testing we found Spark 1.6 a lot
>> more reliable in low memory situations (presumably due to
>> https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
>> probably worth a try.
>>
>> Cheers
>> Ben
>>
>> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>>
>>> Hi,
>>> I would appreciate any clue on this. It has become a bottleneck for our
>>> spark job.
>>>
>>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>>>
 Hi,

 I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
 writing it into Cassandra after processing it. Spark job starts fine and 
 runs all good for some time until I start getting below errors. Once these 
 errors come, job start to lag behind and I see that job has scheduling and 
 processing delays in streaming  UI.

 Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
 memoryFraction parameters. Nothing works.


 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
 curMem=565394, maxMem=2778495713
 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
 bytes in memory (estimated size 3.9 KB, free 2.6 GB)
 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
 took 2 ms
 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
 threshold of 1024.0 KB for computing block broadcast_69652 in memory.
 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
 broadcast_69652 in memory! (computed 496.0 B so far)
 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 
 GB (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 
 GB.
 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
 disk instead.
 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
 452316). 2043 bytes result sent to driver


 Thanks,

 L


>>> --
>> 
>> Ben Slater
>> Chief Product Officer
>> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
>> +61 437 929 798
>>
>
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Gaurav Bhatnagar]

try setting the option --driver-memory 4G

On Tue, Jun 14, 2016 at 3:52 PM, Ben Slater 
wrote:

> A high level shot in the dark but in our testing we found Spark 1.6 a lot
> more reliable in low memory situations (presumably due to
> https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
> probably worth a try.
>
> Cheers
> Ben
>
> On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:
>
>> Hi,
>> I would appreciate any clue on this. It has become a bottleneck for our
>> spark job.
>>
>> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>>
>>> Hi,
>>>
>>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
>>> writing it into Cassandra after processing it. Spark job starts fine and 
>>> runs all good for some time until I start getting below errors. Once these 
>>> errors come, job start to lag behind and I see that job has scheduling and 
>>> processing delays in streaming  UI.
>>>
>>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>>> memoryFraction parameters. Nothing works.
>>>
>>>
>>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>>> curMem=565394, maxMem=2778495713
>>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
>>> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
>>> took 2 ms
>>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>>> broadcast_69652 in memory! (computed 496.0 B so far)
>>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB 
>>> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
>>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to 
>>> disk instead.
>>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
>>> 452316). 2043 bytes result sent to driver
>>>
>>>
>>> Thanks,
>>>
>>> L
>>>
>>>
>> --
> 
> Ben Slater
> Chief Product Officer
> Instaclustr: Cassandra + Spark - Managed | Consulting | Support
> +61 437 929 798
>

Re: Spark Memory Error - Not enough space to cache broadcast

[Ben Slater]

A high level shot in the dark but in our testing we found Spark 1.6 a lot
more reliable in low memory situations (presumably due to
https://issues.apache.org/jira/browse/SPARK-1). If it’s an option,
probably worth a try.

Cheers
Ben

On Wed, 15 Jun 2016 at 08:48 Cassa L  wrote:

> Hi,
> I would appreciate any clue on this. It has become a bottleneck for our
> spark job.
>
> On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:
>
>> Hi,
>>
>> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
>> writing it into Cassandra after processing it. Spark job starts fine and 
>> runs all good for some time until I start getting below errors. Once these 
>> errors come, job start to lag behind and I see that job has scheduling and 
>> processing delays in streaming  UI.
>>
>> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
>> memoryFraction parameters. Nothing works.
>>
>>
>> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
>> curMem=565394, maxMem=2778495713
>> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
>> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
>> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
>> took 2 ms
>> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
>> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
>> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache 
>> broadcast_69652 in memory! (computed 496.0 B so far)
>> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB 
>> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
>> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to disk 
>> instead.
>> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
>> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
>> 452316). 2043 bytes result sent to driver
>>
>>
>> Thanks,
>>
>> L
>>
>>
> --

Ben Slater
Chief Product Officer
Instaclustr: Cassandra + Spark - Managed | Consulting | Support
+61 437 929 798

Re: Spark Memory Error - Not enough space to cache broadcast

[Cassa L]

Hi,
I would appreciate any clue on this. It has become a bottleneck for our
spark job.

On Mon, Jun 13, 2016 at 2:56 PM, Cassa L  wrote:

> Hi,
>
> I'm using spark 1.5.1 version. I am reading data from Kafka into Spark and 
> writing it into Cassandra after processing it. Spark job starts fine and runs 
> all good for some time until I start getting below errors. Once these errors 
> come, job start to lag behind and I see that job has scheduling and 
> processing delays in streaming  UI.
>
> Worker memory is 6GB, executor-memory is 5GB, I also tried to tweak 
> memoryFraction parameters. Nothing works.
>
>
> 16/06/13 21:26:02 INFO MemoryStore: ensureFreeSpace(4044) called with 
> curMem=565394, maxMem=2778495713
> 16/06/13 21:26:02 INFO MemoryStore: Block broadcast_69652_piece0 stored as 
> bytes in memory (estimated size 3.9 KB, free 2.6 GB)
> 16/06/13 21:26:02 INFO TorrentBroadcast: Reading broadcast variable 69652 
> took 2 ms
> 16/06/13 21:26:02 WARN MemoryStore: Failed to reserve initial memory 
> threshold of 1024.0 KB for computing block broadcast_69652 in memory.
> 16/06/13 21:26:02 WARN MemoryStore: Not enough space to cache broadcast_69652 
> in memory! (computed 496.0 B so far)
> 16/06/13 21:26:02 INFO MemoryStore: Memory use = 556.1 KB (blocks) + 2.6 GB 
> (scratch space shared across 0 tasks(s)) = 2.6 GB. Storage limit = 2.6 GB.
> 16/06/13 21:26:02 WARN MemoryStore: Persisting block broadcast_69652 to disk 
> instead.
> 16/06/13 21:26:02 INFO BlockManager: Found block rdd_100761_1 locally
> 16/06/13 21:26:02 INFO Executor: Finished task 0.0 in stage 71577.0 (TID 
> 452316). 2043 bytes result sent to driver
>
>
> Thanks,
>
> L
>
>