nateab commented on PR #27579: URL: https://github.com/apache/flink/pull/27579#issuecomment-5148286684
@rmetzger I dug into the history and the random component was added deliberately in FLINK-7140 (f853f335933): > This should guard us from uploading (and deleting) the same file more than once and also from hash collisions. BlobKey was pure SHA1 before that commit. Going back is hard because there's still no per-blob-key reference counting — PermanentBlobCache counts per JobID only — yet individual PermanentBlobKeys are deleted mid-job via Execution#cleanUpOffloadedTaskRestore and IntermediateResult → BlobServer#deleteInternal, with no check for other referrers. Hash-only keys would collapse byte-identical blobs onto one file and let one owner's cleanup delete it out from under another. The key is also embedded in on-disk/HA-store filenames and the BlobClient wire format. Your question did show the PR was fixing the symptom in the wrong place, though. The mismatch is purely an equality problem in one comparison, so I've reworked it: verifyClassLoader now compares content hashes rather than whole keys. Physical keys stay unique, FLINK-7140's cleanup isolation is untouched, and the exception-driven control flow is gone. Two narrowings I should flag: within that comparison this gives up FLINK-7140's SHA1-collision defense, and it compares the set of distinct contents, so adding/dropping a duplicate of a JAR already on the class path is now accepted. I couldn't restore the old size check faithfully anyway — it compared libraries.size() (a de-duplicated HashSet) against the raw requiredLibraries.size(), so it would already throw on an identical re-registration containing a duplicate key. Happy to reconsider if you'd rather keep it strict. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
