pranavdev022 commented on PR #53914: URL: https://github.com/apache/spark/pull/53914#issuecomment-3784443464
> I think this also fixes the problem of https://github.com/apache/spark/pull/53852 . @cloud-fan, the re-adding of an existing cached local relations resulting in that block dropped was due to the ref-counting logic in ArtifactManager where: - if the same `CacheId(sessionUUID, hash)` block existed, the blockManager skipped adding it. - but, in the ArtifactManager we only checked if same block already exists and then released the block. ``` val blockId = CacheId(sessionUUID, hash) val updater = blockManager.TempFileBasedBlockStoreUpdater(...) updater.save() // in case of already existing block, it skips re-adding but we do not capture the result anywhere val oldBlock = hashToCachedIdMap.put(blockId.hash, new RefCountedCacheId(blockId)) // here, we capture the old refCountedCacheId if (oldBlock != null) { logWarning(...) oldBlock.release(blockManager) // and, released the ONLY occurance of the block } ``` I believe, this issue is different than the race condition issue in IsolatedSessionState? -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected] --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
