Since this first proposal, I was suggested to take a look at credential helper.
With that, I have written a second proposal that uses the credential helper spec for handling credentials. This follows the spec defined in https://github.com/EngFlow/credential-helper-spec/blob/main/spec.md This spec is used by existing tools such as bazel and buildbox: https://github.com/bazelbuild/proposals/blob/main/designs/2022-06-07-bazel-credential-helpers.md https://gitlab.com/BuildGrid/buildbox/buildbox/-/merge_requests/1115 To implement the spec, buildstream should provide: A bst flag --credential-helper <PATH> to specify the credential helper executable BST user config entry auth.credential-helper = "<PATH>" API for plugins to use for retrieving credentials For the credential helper, buildstream should: Timeout after 5 seconds of execution Provide the helper executable the same environment variables as bst Set the helper executable working directory to be the repository root The API would be a utility function get_credentials(uri: str) -> dict[str, Any]: It is a new, isolated function rather than changes to DownloadableFileSource. There are plugin sources that pulls from remote (such as git_repo) that doesn't derive of off DownloadableFileSource The function will attempt authentication with a credential helper, and fallback to netrc. This function would have to sometimes cache the result as specified in the get-credentials response schema: https://github.com/EngFlow/credential-helper-spec/blob/main/schemas/get-credentials-response.schema.json Cache would be stored in a global state Return value is a dictionary containing headers for making requests with. This change will not break existing projects using netrc for authentication. This function should be used within buildstream as well, for anywhere that previously only used netrc for credentials. Feedback is appreciated. If this seems sufficient, I will start creating a patch. Regards, Ed On Wed Jul 8, 2026 at 12:12 PM BST, Ed Cope via dev wrote: > Hello, > > We have a buildstream project that requires us to have two different > credentials under the same domain. > > This doesn't seem possible without modifications to buildstream. > > I have created an issue for this, which can be seen below. > > We are happy to work on an implementation for this, but we want feedback on > if the proposed solution is reasonable. > > Regards, > Ed Cope > > On 2026/07/07 16:31:42 "edcope-ct (via GitHub)" wrote: >> >> edcope-ct opened a new issue, #2148: >> URL: https://github.com/apache/buildstream/issues/2148 >> >> ## Problem >> >> We have a buildstream project that uses sources from private repositories. >> >> These repositories require different credentials, but they are all under >> the same domain. >> This is a problem, as netrc matches domain name but not url paths. >> >> There needs to be a different way to authenticate based on URLs that >> isn't using netrc. >> >> ## Potential solution >> >> URLs could be matched based on globs. Services such as Gitlab lets you >> create tokens for a group, where any repositories under the group can use >> that token to pull the source or LFS tarballs. >> >> These entries could be stored in user configuration, and I suggest >> prioritising authentication with it before using netrc as a fallback. >> This wouldn't create compatibility issues, and allows for per-url >> authentication while still keeping a netrc file. >> >> Example: >> >> ```yaml >> # URL will be matched against each entry. If URL matches an entry, those >> credentials are used. >> authentication: >> # Match repository >> - url: "https://example.com/group1/repo1"; >> username: "user1" >> password: "pass1" >> # Match repository group >> - url: "https://example.com/group1/*"; >> username: "user2" >> password: "pass2" >> # Example of matching Gitlab LFS URL >> # Note that escape sequence is used to escape the ? character >> - url: >> 'https://example.com/api/v4/projects/group1%2Fnestedgroup%2Frepo/repository/files/lfs_object.tar.gz/raw\?ref=*&lfs=true' >> username: "user3" >> password: "user3" >> ``` >> >> ### Additional notes for plugins >> >> This solution would allow for plugins to also authenticate using user >> configuration, without having to implement a custom solution for >> authenticating per-url. >> >> The following merge request was created by me for >> buildstream-plugins-community, where I implement a solution similar to what >> I've described above, but credentials are obtained from `~/.gitconfig`: >> >> >> https://gitlab.com/BuildStream/buildstream-plugins-community/-/merge_requests/445. >> >> >> -- >> This is an automated message from the Apache Git Service. >> To respond to the message, please log on to GitHub and use the >> URL above to go to the specific comment. >> >> To unsubscribe, e-mail: [email protected] >> >> For queries about this service, please contact Infrastructure at: >> [email protected] >> >>
