Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Fri, May 6, 2022 at 12:17 PM Marián Konček wrote:
>
> I agree with Vit, I want to add to this:
>
> * still having to maintain a custom script is not an improvement over our
> current situation.
> * make sure that $VERSION can be processed, there are cases when for example
> the tilde "~" needs to be swapped for dash "-".
> * More complex case of $VERSION: obtain a specific commit from upstream,
> example:
> https://src.fedoraproject.org/fork/mkoncek/rpms/javapackages-bootstrap/blob/rawhide/f/javapackages-bootstrap.spec#_19
>
> The only thing that needs to happen in this phase is removing files.
> Therefore there is no need to allow the flexibility like in the %prep stage.
> The whole function should look like this:
>
> `obtain_sources $processed_version $list_of_patterns_to_exclude`
>
> where $processed_version as well as $list_of_patterns_to_exclude can be
> written / computed directly in the .spec file.
>
> Like in
> "https://github.com/fedora-java/javapackages-bootstrap/archive/%{git_short_hash}.tar.gz;
> For $list_of_patterns_to_exclude, we generally want to exclude either
> specific files / directories or names based on their prefixes / suffixes.
> Basically we need the functionality of chained `rm` and `find ... -delete`.
Isn't this starting to move goal-posts? I provided a draft
implementation that solves 100% of the problem stated in this thread:
A way to automate fetching / preparing arbitrary sources that is
compatible and integrated with existing packaging workflows, and
should be compatible with the CI stuff we have now.
If you want something that is not a separate config / script file but
embedded inside the spec file, you'd probably need to file an RFE with
RPM for that, because currently you don't have enough access to RPM
.spec parsing machinery to implement something like this, as far as I
can tell.
And even if you want to throw away all currently existing scripts and
use something that's a limited, 100% declarative solution, you'd still
need to integrate this into current packaging workflows somehow
(probably in spectool). But spectool uses RPM python bindings to parse
the .spec file, and that doesn't expose stuff like the values of
macros, so you're back to either filing an RFE with RPM for adding
APIs for that, or using an external config file.
If you prefer a declarative approach using an external config file
(instead of an external script) for spectool, I can work on that.
Heck, I've been using a very similar tool to automate my COPR with
nightly builds of Pantheon / elementary OS components for over half a
decade.
But the thing you seem to want (both *100% declarative* and
*integrated in the .spec file*) is not currently possible, as far as I
can tell (unless you start parsing .spec files yourself, which - trust
me - is not something you want to do).
Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
I agree with Vit, I want to add to this:
* still having to maintain a custom script is not an improvement over
our current situation.
* make sure that $VERSION can be processed, there are cases when for
example the tilde "~" needs to be swapped for dash "-".
* More complex case of $VERSION: obtain a specific commit from upstream,
example:
https://src.fedoraproject.org/fork/mkoncek/rpms/javapackages-bootstrap/blob/rawhide/f/javapackages-bootstrap.spec#_19
The only thing that needs to happen in this phase is removing files.
Therefore there is no need to allow the flexibility like in the %prep
stage. The whole function should look like this:
`obtain_sources $processed_version $list_of_patterns_to_exclude`
where $processed_version as well as $list_of_patterns_to_exclude can be
written / computed directly in the .spec file.
Like in
"|https://github.com/fedora-java/javapackages-bootstrap/archive/%{git_short_hash}.tar.gz|"
For $list_of_patterns_to_exclude, we generally want to exclude either
specific files / directories or names based on their prefixes /
suffixes. Basically we need the functionality of chained `rm` and `find
... -delete`.
On 6. 5. 2022 11:36, Vít Ondruch wrote:
Dne 05. 05. 22 v 19:35 Fabio Valentini napsal(a):
On Tue, Apr 26, 2022 at 1:50 PM Marián Konček
wrote:
Adding onto this, as I mentioned an additional step, I can also imagine
defining some macros which would be used in the .spec file. We could
have a simple generic one that would be available within rpm with the
extension possibility of each working group defining their own macros
which they would use in their own ecosystems.
Not everything has to be an RPM macro ... And I'm not even sure how
you would extract the value of these macros from the .spec file for
external consumption. Parsing RPM .spec files yourself is very hard,
and RPM's own spec parsing machinery doesn't really expose all that
much of its innards to the outside via its API.
I've spent a few minutes working on a DRAFT of my suggested
"SourceScript"-based functionality for spectool here:
https://pagure.io/rpmdevtools/pull-request/94
It was surprisingly simple to implement, works just fine, and
integrates with the default "spectool -g *.spec" use case.
It just calls the specified script with the value of the .spec file's
"Version" tag as an argument (making it unnecessary to parse the .spec
file in the script for a second time, just to extract the version
string again). Adding other CLI arguments (though I wouldn't know any
argument other than "%{version}" that would be useful here) would be
simple.
I'd be interested to see whether that would be enough to satisfy the
use case mentioned at the start of the thread, though I think the
current implementation should be flexible enough (maybe too flexible,
even).
Sorry, but I am not happy about that, because
1) the script is external to the .spec file
2) there still needs to be related `Source10: path-to-script.sh` to
have the script included in SRPM.
Vít
Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report
it:https://pagure.io/fedora-infrastructure
--
Marián Konček
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
Dne 05. 05. 22 v 19:35 Fabio Valentini napsal(a):
On Tue, Apr 26, 2022 at 1:50 PM Marián Konček wrote:
Adding onto this, as I mentioned an additional step, I can also imagine
defining some macros which would be used in the .spec file. We could
have a simple generic one that would be available within rpm with the
extension possibility of each working group defining their own macros
which they would use in their own ecosystems.
Not everything has to be an RPM macro ... And I'm not even sure how
you would extract the value of these macros from the .spec file for
external consumption. Parsing RPM .spec files yourself is very hard,
and RPM's own spec parsing machinery doesn't really expose all that
much of its innards to the outside via its API.
I've spent a few minutes working on a DRAFT of my suggested
"SourceScript"-based functionality for spectool here:
https://pagure.io/rpmdevtools/pull-request/94
It was surprisingly simple to implement, works just fine, and
integrates with the default "spectool -g *.spec" use case.
It just calls the specified script with the value of the .spec file's
"Version" tag as an argument (making it unnecessary to parse the .spec
file in the script for a second time, just to extract the version
string again). Adding other CLI arguments (though I wouldn't know any
argument other than "%{version}" that would be useful here) would be
simple.
I'd be interested to see whether that would be enough to satisfy the
use case mentioned at the start of the thread, though I think the
current implementation should be flexible enough (maybe too flexible,
even).
Sorry, but I am not happy about that, because
1) the script is external to the .spec file
2) there still needs to be related `Source10: path-to-script.sh` to have
the script included in SRPM.
Vít
Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Tue, Apr 26, 2022 at 1:50 PM Marián Konček wrote:
>
> Adding onto this, as I mentioned an additional step, I can also imagine
> defining some macros which would be used in the .spec file. We could
> have a simple generic one that would be available within rpm with the
> extension possibility of each working group defining their own macros
> which they would use in their own ecosystems.
Not everything has to be an RPM macro ... And I'm not even sure how
you would extract the value of these macros from the .spec file for
external consumption. Parsing RPM .spec files yourself is very hard,
and RPM's own spec parsing machinery doesn't really expose all that
much of its innards to the outside via its API.
I've spent a few minutes working on a DRAFT of my suggested
"SourceScript"-based functionality for spectool here:
https://pagure.io/rpmdevtools/pull-request/94
It was surprisingly simple to implement, works just fine, and
integrates with the default "spectool -g *.spec" use case.
It just calls the specified script with the value of the .spec file's
"Version" tag as an argument (making it unnecessary to parse the .spec
file in the script for a second time, just to extract the version
string again). Adding other CLI arguments (though I wouldn't know any
argument other than "%{version}" that would be useful here) would be
simple.
I'd be interested to see whether that would be enough to satisfy the
use case mentioned at the start of the thread, though I think the
current implementation should be flexible enough (maybe too flexible,
even).
Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
Dne 26. 04. 22 v 12:40 Vít Ondruch napsal(a): Dne 25. 04. 22 v 10:41 Vít Ondruch napsal(a): We already have signature of source files as additional source. Do you mean the 'sources' file? I mean https://docs.fedoraproject.org/en-US/packaging-guidelines/#_verifying_signatures Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
Adding onto this, as I mentioned an additional step, I can also imagine defining some macros which would be used in the .spec file. We could have a simple generic one that would be available within rpm with the extension possibility of each working group defining their own macros which they would use in their own ecosystems. Debian has an article about reproducible packaging: https://wiki.debian.org/ReproducibleBuilds/Howto#Introduction Especially look for the string: "tarball" -- Marián Konček ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
> Dne 25. 04. 22 v 10:41 Vít Ondruch napsal(a): > > We already have signature of source files as additional source. Do you mean the 'sources' file? 1) This is dist-git feature, external to RPM, which is quite unfortunate. 2) There are certainly requests to include this into .spec file: https://github.com/rpm-software-management/rpm/issues/463 > > But the additional section as suggested by Marián works for me too. > > I just want to take the script and paste it somewhere without worrying about > escaping or > wrapping. But the point is that you don't have to copy paste this, because RPM executes the scriptlet for you. Actually, I just lerned about `%_disable_source_fetch` flag, this could be reused for this fucntionality. Vít ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
> On Mon, Apr 25, 2022 at 10:51 AM Vít Ondruch I also assume that the CI you're talking about already calls spectool > to download package sources for new versions I don't think that Koji simple CI nor Zuul does this. BTW sorry, I just noticed that I have not received ~5 mail from this thread. I don't know where the problem is, if on mailman on RH email infrastructure side. Vít ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Tue, Apr 26, 2022 at 11:49 AM Fabio Valentini wrote: > > On Tue, Apr 26, 2022 at 10:08 AM David Bold > wrote: > > > > > I think it does require some changes to CI, otherwise, this will execute > > untrusted code when all it was supposed to do is download. I do > > currently assume that I can run `spectool -g` on an untrusted spec to > > look at the source code, without running untrusted code. Also, this sounds like you misunderstand what spectool does. All it does is call RPM to parse the .spec file (which might already execute arbitrary lua code), and then get the list of Source and Patch files from that parsed data. If you specify the "-g" flag, it will also attempt to download those files, if they are specified as URLs. If you want to do something else, you're probably looking for a different tool. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Tue, Apr 26, 2022 at 10:08 AM David Bold wrote: > > I think it does require some changes to CI, otherwise, this will execute > untrusted code when all it was supposed to do is download. I do > currently assume that I can run `spectool -g` on an untrusted spec to > look at the source code, without running untrusted code. Unless you want to propose an entirely declarative way to specify how to generate "clean" tarballs, you're going to end up executing user code. > I think the user should make an active decision to execute such scripts, > thus a API change for CI would be needed. If you don't want user code to be executed at all, you're already fighting a losing battle against RPM. All .spec files and macros can contain arbitrary lua scripts, which are executed at parse-time. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On 4/25/22 13:42, Fabio Valentini wrote: On Mon, Apr 25, 2022 at 10:51 AM Vít Ondruch wrote: 2) Standalone script does not solve the main issue and that is a way CI could obtain the tarball. Of course you mentioned "with support for extraction in spectool", but that is also part of the issue, because that would need the "spectool" changes as well as CI changes. My proposal does not need that. Of course, this is proof of concept, while the part of the script you point out could be possibly improved and abstracted by some macro. This has come up before, but given that the current maintainer of spectool (which is me) has offered to implement support for this, I don't see this as a problem. I also assume that the CI you're talking about already calls spectool to download package sources for new versions, so doing this would actually make any changes to the CI environment entirely unnecessary. We'd just need to agree on a way to specify the path to the script that needs to be run for generating source X. For example, we could use something like: # SourceScript: gen_clean_tarball.sh That would make it easy for spectool to parse this information from the .spec file, and then execute the program with that name. I think it does require some changes to CI, otherwise, this will execute untrusted code when all it was supposed to do is download. I do currently assume that I can run `spectool -g` on an untrusted spec to look at the source code, without running untrusted code. I think the user should make an active decision to execute such scripts, thus a API change for CI would be needed. David We could possibly also supply some variables as command line arguments to that script, for example, the current "Version" from the .spec file, so it doesn't have to be modified in two places. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On 25/04/2022 14:54, Fabio Valentini wrote: How would the program know if this Source is one that should be executed, or not? We can execute SourceX files with chmod 0755 and .sh extension. Which is why I suggested to use a marker comment like "# SourceScript: foo.sh". It will not be included into SRPM. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
Dne 25. 04. 22 v 10:41 Vít Ondruch napsal(a): 1) Standalone script is kind of against RPM philosophy, where the idea always was that the .spec file should contain everything. We already have signature of source files as additional source. But the additional section as suggested by Marián works for me too. I just want to take the script and paste it somewhere without worrying about escaping or wrapping. Miroslav ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
+1 to using an rpm macro to avoid adding an external script, if spectool can work with it. Something like: %global source0_generate_script ( \ curl ... \ rm -rf ... \ tar ... ) I'm not sure if that syntax is correct.___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Mon, Apr 25, 2022 at 2:46 PM Vitaly Zaitsev via devel
wrote:
>
> On 25/04/2022 13:42, Fabio Valentini wrote:
> > For example, we could use something like:
> > # SourceScript: gen_clean_tarball.sh
>
> Source1: gen_clean_tarball.sh
Well, doing that alone wouldn't work. How would the program know if
this Source is one that should be executed, or not?
So, even if the script is included as a Source file for the SRPM, we'd
still need to specify *which* file to actually execute (if any).
Which is why I suggested to use a marker comment like "# SourceScript: foo.sh".
> > That would make it easy for spectool to parse this information from
> > the .spec file, and then execute the program with that name.
>
> Can be easily parsed even with regular expression.
Sure. script_path = if line.startswith("# SourceScript:"): line.split(" \t")[1]
That's not a problem. But we still need to specify *which* source file
is actually the script to run (if any).
Fabio
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Mon, Apr 25, 2022 at 7:51 AM Fabio Valentini wrote: > > On Mon, Apr 25, 2022 at 10:51 AM Vít Ondruch wrote: > > > 2) Standalone script does not solve the main issue and that is a way CI > > could obtain the tarball. Of course you mentioned "with support for > > extraction in spectool", but that is also part of the issue, because that > > would need the "spectool" changes as well as CI changes. My proposal does > > not need that. Of course, this is proof of concept, while the part of the > > script you point out could be possibly improved and abstracted by some > > macro. > > This has come up before, but given that the current maintainer of > spectool (which is me) has offered to implement support for this, I > don't see this as a problem. > I also assume that the CI you're talking about already calls spectool > to download package sources for new versions, so doing this would > actually make any changes to the CI environment entirely unnecessary. > > We'd just need to agree on a way to specify the path to the script > that needs to be run for generating source X. > For example, we could use something like: > # SourceScript: gen_clean_tarball.sh > > That would make it easy for spectool to parse this information from > the .spec file, and then execute the program with that name. > We could possibly also supply some variables as command line arguments > to that script, for example, the current "Version" from the .spec > file, so it doesn't have to be modified in two places. > The "easy" thing to do would be to define a config file to check into Dist-Git to run source services. We could even reuse the ones that the openSUSE community wrote for the Open Build Service. I could easily write a harness to run them inside the Koji task for producing the SRPM. I already did something similar for my workplace, for example. -- 真実はいつも一つ！/ Always, there's only one truth! ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On 25/04/2022 13:42, Fabio Valentini wrote: For example, we could use something like: # SourceScript: gen_clean_tarball.sh Source1: gen_clean_tarball.sh That would make it easy for spectool to parse this information from the .spec file, and then execute the program with that name. Can be easily parsed even with regular expression. -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On Mon, Apr 25, 2022 at 10:51 AM Vít Ondruch wrote: > 2) Standalone script does not solve the main issue and that is a way CI could > obtain the tarball. Of course you mentioned "with support for extraction in > spectool", but that is also part of the issue, because that would need the > "spectool" changes as well as CI changes. My proposal does not need that. Of > course, this is proof of concept, while the part of the script you point out > could be possibly improved and abstracted by some macro. This has come up before, but given that the current maintainer of spectool (which is me) has offered to implement support for this, I don't see this as a problem. I also assume that the CI you're talking about already calls spectool to download package sources for new versions, so doing this would actually make any changes to the CI environment entirely unnecessary. We'd just need to agree on a way to specify the path to the script that needs to be run for generating source X. For example, we could use something like: # SourceScript: gen_clean_tarball.sh That would make it easy for spectool to parse this information from the .spec file, and then execute the program with that name. We could possibly also supply some variables as command line arguments to that script, for example, the current "Version" from the .spec file, so it doesn't have to be modified in two places. Fabio ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
I would also be interested in this. In our set of java packages we have
a few cases where upstream releases .jars with bundled binary files
which either cannot be stored within .src.rpm files because of licensing
issues or because they would be unreasonably large. Example here:
https://src.fedoraproject.org/rpms/byte-buddy/blob/rawhide/f/generate-tarball.sh
Our scripts are mostly copy-paste with some additional removals where
needed. This is in addition to the %prep step where we tend to remove
some parts as well. I never questioned this approach but I agree that I
would like to have it automated.
When repackaging, there are issues that need to be addressed and decided
whether they pose a real problem like file attributes, file sorting,
timestamps for example.
It looks like we would want some %pre-prep step :)
On 25. 4. 2022 10:41, Vít Ondruch wrote:
Dne 21. 04. 22 v 14:58 Miroslav Suchý napsal(a):
Dne 21. 04. 22 v 13:20 Vít Ondruch napsal(a):
Now I am looking for feedback about general approach. Of course it
could be somehow polished and improved to hide some boiler plate.
This part:
%{echo:%(
[ ! -e %{S:1} ] &&
Looks really clumsy. After reading the
https://pagure.io/packaging-committee/issue/1132#comment-769233
I like
# Script: gen_clean_tarball.sh
approach. Whether it will be special comment, macro (with support for
extraction in spectool) or new tag in RPM - I do not care. The
important part is that it is standalone file, which can be easily
locally executed. That would ease development and debugging.
1) Standalone script is kind of against RPM philosophy, where the idea
always was that the .spec file should contain everything.
2) Standalone script does not solve the main issue and that is a way
CI could obtain the tarball. Of course you mentioned "with support for
extraction in spectool", but that is also part of the issue, because
that would need the "spectool" changes as well as CI changes. My
proposal does not need that. Of course, this is proof of concept,
while the part of the script you point out could be possibly improved
and abstracted by some macro.
Vít
Miroslav
___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report
it:https://pagure.io/fedora-infrastructure
___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report
it:https://pagure.io/fedora-infrastructure
--
Marián Konček
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
Dne 21. 04. 22 v 14:58 Miroslav Suchý napsal(a):
Dne 21. 04. 22 v 13:20 Vít Ondruch napsal(a):
Now I am looking for feedback about general approach. Of course it
could be somehow polished and improved to hide some boiler plate.
This part:
%{echo:%(
[ ! -e %{S:1} ] &&
Looks really clumsy. After reading the
https://pagure.io/packaging-committee/issue/1132#comment-769233
I like
# Script: gen_clean_tarball.sh
approach. Whether it will be special comment, macro (with support for
extraction in spectool) or new tag in RPM - I do not care. The
important part is that it is standalone file, which can be easily
locally executed. That would ease development and debugging.
1) Standalone script is kind of against RPM philosophy, where the idea
always was that the .spec file should contain everything.
2) Standalone script does not solve the main issue and that is a way CI
could obtain the tarball. Of course you mentioned "with support for
extraction in spectool", but that is also part of the issue, because
that would need the "spectool" changes as well as CI changes. My
proposal does not need that. Of course, this is proof of concept, while
the part of the script you point out could be possibly improved and
abstracted by some macro.
Vít
Miroslav
___
devel mailing list --devel@lists.fedoraproject.org
To unsubscribe send an email todevel-le...@lists.fedoraproject.org
Fedora Code of
Conduct:https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines:https://fedoraproject.org/wiki/Mailing_list_guidelines
List
Archives:https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report
it:https://pagure.io/fedora-infrastructure
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
On 21/04/2022 13:20, Vít Ondruch wrote: It is quite common, to have some sources, which are not available as a tarball from upstream. In case of rubygem- packages, they quite often do not ship their test suites. In this case, our .spec file contains something like [1]: I prefer a separate make_tarball.sh script. It must be added as Source. Example: https://src.fedoraproject.org/rpms/hashcat/blob/rawhide/f/hashcat.spec#_30 -- Sincerely, Vitaly Zaitsev (vit...@easycoding.org) ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org Do not reply to spam on the list, report it: https://pagure.io/fedora-infrastructure
Re: A way to prepare custom source tarballs from .spec file to improve CI experience
Dne 21. 04. 22 v 13:20 Vít Ondruch napsal(a):
Now I am looking for feedback about general approach. Of course it could be somehow polished and improved to hide some
boiler plate.
This part:
%{echo:%(
[ ! -e %{S:1} ] &&
Looks really clumsy. After reading the
https://pagure.io/packaging-committee/issue/1132#comment-769233
I like
# Script: gen_clean_tarball.sh
approach. Whether it will be special comment, macro (with support for extraction in spectool) or new tag in RPM - I do
not care. The important part is that it is standalone file, which can be easily locally executed. That would ease
development and debugging.
Miroslav
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
A way to prepare custom source tarballs from .spec file to improve CI experience
Hi,
It is quite common, to have some sources, which are not available as a
tarball from upstream. In case of rubygem- packages, they quite often do
not ship their test suites. In this case, our .spec file contains
something like [1]:
~~~
# Tests are not packaged with the gem. You may get them like so:
# git clone --no-checkout https://github.com/discourse/mini_mime
# git -C mini_mime archive -v -o mini_mime-1.1.0-test.txz v1.1.0 test
Source1: %{gem_name}-%{version}-test.txz
~~~
This is quite fine. However, if there is opened PR [2], the CI would
work only as long sources are in the look a side cache, which is not
always desirable, because this might be just some WIP.
I while ago, I proposed [3] this "executable" comments instead:
~~~
Source1: %{gem_name}-%{version}-test.txz
%{echo:%(
[ ! -e %{S:1} ] &&
rm -rf %{name} &&
git clone https://github.com/discourse/mini_mime %{name} &&
git -C %{name} archive -v -o %{S:1} v%{version} test/
)}
~~~
I know, that some may say "it won't work in Koji", that is true, but
won't necessarily be issue for CI. The other argument might be "it is
creating some random tarball from random source", but if the accompanied
`sources` file contains the right checksums, `fedpkg srpm` ensures that
only the tarballs with the right content is used.
While others might just handwavy upload the random sources into look a
side cache, I think this is way better approach.
Now I am looking for feedback about general approach. Of course it could
be somehow polished and improved to hide some boiler plate.
Vít
[1]:
https://src.fedoraproject.org/rpms/rubygem-mini_mime/blob/c25611d64d17c374aa274391784ab9ff5fe00da8/f/rubygem-mini_mime.spec#_14
[2]: https://src.fedoraproject.org/rpms/rubygem-mini_mime/pull-request/2
[3]: https://pagure.io/packaging-committee/issue/1132#comment-769233
OpenPGP_signature
Description: OpenPGP digital signature
___
devel mailing list -- devel@lists.fedoraproject.org
To unsubscribe send an email to devel-le...@lists.fedoraproject.org
Fedora Code of Conduct:
https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives:
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Do not reply to spam on the list, report it:
https://pagure.io/fedora-infrastructure
