attilapiros commented on issue #24554: [SPARK-27622][Core] Avoiding the network when block manager fetches disk persisted RDD blocks from the same host URL: https://github.com/apache/spark/pull/24554#issuecomment-491284321 Thanks @squito! I will look into the problem regarding unpersist and file delete. On linux what I have found is already promising: https://linux.die.net/man/2/unlink : >If the name was the last link to a file but any processes still have the file open the file will remain in existence until the last file descriptor referring to it is closed. Of course I will test this and come back to you with the result. Moreover as I know on windows a file cannot be deleted if it is opened. But I really have to check the details. I have not tried this PR but I will do that too.
---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: [email protected] With regards, Apache Git Services --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
