Re: test failed due to OOME

I noticed that the SparkContext created in each sub-test is not stopped upon finishing sub-test. Would stopping each SparkContext make a difference in terms of heap memory consumption ? Cheers On Fri, Oct 30, 2015 at 12:04 PM, Mridul Muralidharan wrote: > It is giving OOM at

Re: Maintaining overall cumulative data in Spark Streaming

How to we reset the aggregated statistics to null? Regards, Sandeep Giri, +1 347 781 4573 (US) +91-953-899-8962 (IN) www.KnowBigData.com. Phone: +1-253-397-1945 (Office) [image: linkedin icon] [image: other site icon]

Re: Re: repartitionAndSortWithinPartitions task shuffle phase is very slow

Would love to have any suggestion or comments about our implementation. Is there anyone who has such experience? Thanks. Best Regards! - Luke Han On Tue, Oct 27, 2015 at 10:33 AM, 周千昊 wrote: > I have replace default java serialization with Kyro. >

Off-heap storage and dynamic allocation

Hey guys, According to the docs for 1.5.1, when an executor is removed for dynamic allocation, the cached data is gone. If I use off-heap storage like tachyon, conceptually there isn't this issue anymore, but is the cached data still available in practice? This would be great because then we