kotborealis opened a new issue, #2120: URL: https://github.com/apache/buildstream/issues/2120
While using buildstream I found the following issue: ``` [--:--:--] STATUS [59065303] self_junction_a.bst:example.bst: Pulling data for artifact 59065303 <- https://... [--:--:--] INFO [59065303] self_junction_a.bst:example.bst: Pulled artifact 59065303 <- https://... [--:--:--] INFO [59065303] self_junction_b.bst:example.bst: Pulled artifact 59065303 <- https://... [00:00:13] SUCCESS [59065303] self_junction_a.bst:example.bst: Pull [00:00:14] FAILURE [59065303] self_junction_b.bst:example.bst: Failed to link '.../bst-cache/artifacts/refs/project/example/590653033f888793cebe4368a90d0cce91bdacb468fc29ba369cc16449888189 -> .../bst-cache/artifacts/refs/project/example/f4f620fd79b8ad9b0d271c4bd18bb00483b6d92614dc3b2aec8eac049ae4df7f': [Errno 17] File exists: '.../bst-cache/artifacts/refs/project/example/590653033f888793cebe4368a90d0cce91bdacb468fc29ba369cc16449888189' -> '.../bst-cache/artifacts/refs/project/example/f4f620fd79b8ad9b0d271c4bd18bb00483b6d92614dc3b2aec8eac049ae4df7f' ``` In my project I use multiple self-junctions - junctions that load the project itself to build it differently or to overwrite some elements. While pulling artifacts in parallel for different self-junctions from cache there's this issue `Failed to link ... File exists`. I cannot reproduce this error locally, only on specific CI. I found mentions of the same error in existing issues/PRs: * https://github.com/apache/buildstream/issues/1740 * https://github.com/apache/buildstream/pull/1881#issuecomment-1837236850 * https://github.com/apache/buildstream/pull/1820#issuecomment-1426807683 My guess is that its some race caused by [`safe_link`](https://github.com/apache/buildstream/blob/4c92f3003cfba379edd9d653a72d860ded013445/src/buildstream/utils.py#L387-L398) function. I'm trying to produce some fix for this race, what should I consider while tinkering with `safe_link`? My best bet is to make link operation more atomic by first making hard-link in some temp directory and them moving it to original destination. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
