Thanks for bringing this up. Wouldn't it be better for the notebooks to control when these DFs/RDDs expire so they can do fine granular control?
On Wed, Oct 16, 2024 at 7:51 AM Holden Karau <holden.ka...@gmail.com> wrote: > Hi Spark Devs, > > So back in Spark 1.X we had shuffle TTLs, but they did not take into > account last access times. With the increased use of notebooks where > dataframes & rdds are more likely to be defined at the global scope I was > thinking it could be a good time to try and re-introduce shuffle TTLs but > with a last accessed mechanism so I've filed > https://issues.apache.org/jira/browse/SPARK-49788 -- I'd love to get > folks feedback before I put in too much effort here. > > Cheers, > > Holden :) > > -- > Twitter: https://twitter.com/holdenkarau > Fight Health Insurance: https://www.fighthealthinsurance.com/ > <https://www.fighthealthinsurance.com/?q=hk_email> > Books (Learning Spark, High Performance Spark, etc.): > https://amzn.to/2MaRAG9 <https://amzn.to/2MaRAG9> > YouTube Live Streams: https://www.youtube.com/user/holdenkarau > Pronouns: she/her >
