Removing Data Spark automatically monitors cache usage on each node and drops out old data partitions in a least-recently-used (LRU) fashion. If you would like to manually remove an RDD instead of waiting for it to fall out of the cache, use the RDD.unpersist() method. (Copied from Documentation<http://spark.apache.org/docs/latest/programming-guide.html#removing-data> )
From: Cesar [mailto:ces...@gmail.com] Sent: 31 August 2016 16:57 To: user Subject: releasing memory without stopping the spark context ? Is there a way to release all persisted RDD's/DataFrame's in Spark without stopping the SparkContext ? Thanks a lot -- Cesar Flores This e-mail and any attachments are confidential, intended only for the addressee and may be privileged. If you have received this e-mail in error, please notify the sender immediately and delete it. Any content that does not relate to the business of Worldpay is personal to the sender and not authorised or endorsed by Worldpay. Worldpay does not accept responsibility for viruses or any loss or damage arising from transmission or access. Worldpay (UK) Limited (Company No: 07316500/ Financial Conduct Authority No: 530923), Worldpay Limited (Company No:03424752 / Financial Conduct Authority No: 504504), Worldpay AP Limited (Company No: 05593466 / Financial Conduct Authority No: 502597). Registered Office: The Walbrook Building, 25 Walbrook, London EC4N 8AF and authorised by the Financial Conduct Authority under the Payment Service Regulations 2009 for the provision of payment services. Worldpay (UK) Limited is authorised and regulated by the Financial Conduct Authority for consumer credit activities. Worldpay B.V. (WPBV) has its registered office in Amsterdam, the Netherlands (Handelsregister KvK no. 60494344). WPBV holds a licence from and is included in the register kept by De Nederlandsche Bank, which registration can be consulted through www.dnb.nl. Worldpay, the logo and any associated brand names are trade marks of the Worldpay group.
