Re: Sourcegraph and exploded sources
On Sat, Nov 20, 2021 at 04:19:33PM -0500, Stephen John Smoogen wrote: > 0. When you mean %prep and %build you mean where koji builds a src.rpm > and then the builders get it versus the prep/build stage of mock? I mean in between those stages in rpmbuild. > 1. I don't think the builders don't have access to the internet to do > that. Koji would have access but it is a limited resource. Oh, yeah. I knew that -- I was thinking we could make an outgoing rule to allow this access, but the idea of using gitlab would mean that's a pretty big hole. So I guess a better process would be to have a separate service which would 1. follow the message bus for successful non-scratch builds 2. grab the src rpm from koji 3. run `mock --short-circuit prep $SOURCERPM` 4. grab the results from mock's root/builddir/build/BUILD directory 5+ (all the stuff with git) > 2. I don't think you want any and every build to be broken because we > could not get to gitlab for a push due to timeouts, problems with > snips in the code etc. I was thinking it could just warn on failure, not block. But yeah. > You would want to plumb this into the build process as a completely > new tool. It would need a dedicated box which does this and it would > need to be able to > a) not stop builds and composes > b) be able to queue these actions so a mass rebuild doesn't take weeks > c) be able to resend/redo when we hit gitlab max actions per > second/hour. [Even paid accounts have quotas to keep overall service > working] > d) deal with MBS issues. *nod* -- Matthew Miller Fedora Project Leader ___ infrastructure mailing list -- infrastructure@lists.fedoraproject.org To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: Sourcegraph and exploded sources
On Sat, 20 Nov 2021 at 15:05, Matthew Miller wrote:
>
> I mentioned on devel list that Sourcegraph is going to be indexing
> source.fedoraproject.org.
>
> So I guess, first — heads-up! if you see their traffic, it's not malicious.
> (I asked them to provide us with the user agent they use, and I'll pass that
> on.)
>
> But then, I'd actually like to go a step further, and have them index _the
> actual source for every build_. They're open to doing that, but what they
> need is a git repo.
>
> So.. how hard / ridiculous / bad would it be to have a step in the build
> process in koji which, between the %prep and %build phases, push the
> unpacked-and-prepped source tree to Gitlab, under, say,
> https://gitlab.com/fedora/exploded-build-sources/package-name/?
>
0. When you mean %prep and %build you mean where koji builds a src.rpm
and then the builders get it versus the prep/build stage of mock?
1. I don't think the builders don't have access to the internet to do
that. Koji would have access but it is a limited resource.
2. I don't think you want any and every build to be broken because we
could not get to gitlab for a push due to timeouts, problems with
snips in the code etc.
You would want to plumb this into the build process as a completely
new tool. It would need a dedicated box which does this and it would
need to be able to
a) not stop builds and composes
b) be able to queue these actions so a mass rebuild doesn't take weeks
c) be able to resend/redo when we hit gitlab max actions per
second/hour. [Even paid accounts have quotas to keep overall service
working]
d) deal with MBS issues.
Otherwise you are going to run into various limitations that could
break the koji+mbs+pdc+bodhi+composer+etc which are all built with
certain tolerances and break when something slows things down.
> I'm imaginging this would work something like this:
>
> 1. If remote repo doesn't exist, create it with the gitlab api
>
> 2. Do a shallow, no-checkout clone of that remote repo, using
>--git-dir and --work-tree so that the .git directory isn't created inside
>the working directory
>
> 3. copy the unpacked source tree from %prep into the work-tree dir
>(are we building on btrfs? cp -l if not, to save io?)
>
> 4. git add --all
>
> 5. git commit -m "Build ID: ${buildID}
> https://koji.fedoraproject.org/koji/buildinfo?buildID=${buildID};
>
> 6. git push
>
>
> With some details to be worked out. :) Like, repo tags as branches, maybe?
> And, do this on all arches, or just one of them? Or maybe run up through
> %prep as part of the src build rather than any of the binary builds? Make
> the commit as the Fedora username of the person who did the build?
>
> But those kinds of implementation details aside -- does this seem like
> something we might be able to do?
>
> Or, any ideas for an alternate approach? I mean, obviously, source-git would
> be one such alternative, but getting that for _everything_ would require a
> big rework of how everyone works. (I think we should get to that eventually,
> but that's a long way off.)
>
> --
> Matthew Miller
>
> Fedora Project Leader
> ___
> infrastructure mailing list -- infrastructure@lists.fedoraproject.org
> To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
> Fedora Code of Conduct:
> https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives:
> https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
> Do not reply to spam on the list, report it:
> https://pagure.io/fedora-infrastructure
--
Stephen J Smoogen.
Let us be kind to one another, for most of us are fighting a hard
battle. -- Ian MacClaren
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Sourcegraph and exploded sources
I mentioned on devel list that Sourcegraph is going to be indexing
source.fedoraproject.org.
So I guess, first — heads-up! if you see their traffic, it's not malicious.
(I asked them to provide us with the user agent they use, and I'll pass that
on.)
But then, I'd actually like to go a step further, and have them index _the
actual source for every build_. They're open to doing that, but what they
need is a git repo.
So.. how hard / ridiculous / bad would it be to have a step in the build
process in koji which, between the %prep and %build phases, push the
unpacked-and-prepped source tree to Gitlab, under, say,
https://gitlab.com/fedora/exploded-build-sources/package-name/?
I'm imaginging this would work something like this:
1. If remote repo doesn't exist, create it with the gitlab api
2. Do a shallow, no-checkout clone of that remote repo, using
--git-dir and --work-tree so that the .git directory isn't created inside
the working directory
3. copy the unpacked source tree from %prep into the work-tree dir
(are we building on btrfs? cp -l if not, to save io?)
4. git add --all
5. git commit -m "Build ID: ${buildID}
https://koji.fedoraproject.org/koji/buildinfo?buildID=${buildID};
6. git push
With some details to be worked out. :) Like, repo tags as branches, maybe?
And, do this on all arches, or just one of them? Or maybe run up through
%prep as part of the src build rather than any of the binary builds? Make
the commit as the Fedora username of the person who did the build?
But those kinds of implementation details aside -- does this seem like
something we might be able to do?
Or, any ideas for an alternate approach? I mean, obviously, source-git would
be one such alternative, but getting that for _everything_ would require a
big rework of how everyone works. (I think we should get to that eventually,
but that's a long way off.)
--
Matthew Miller
Fedora Project Leader
___
infrastructure mailing list -- infrastructure@lists.fedoraproject.org
To unsubscribe send an email to infrastructure-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/infrastructure@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
