Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Removing unneeded internal macros (such as %__ranlib) (#1211)
> For no real value, yes but still they do are used in the wild and so the > potential for breakage is quite wide… Of course, but it is not different from changing other behaviors of RPM. Those macros are not documented, they are explicitly described as *private to RPM* so I think making such change in next version of RPM (4.17) can be affordable. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1211#issuecomment-626291865___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
So I guess this is waiting for me to put my thoughts here…
# Features (extras)
* In Rust, `Cargo.toml` contains information about all "features" which should
be in their own subpackages, like `%package devel+$FEATURE`.
* In Python, `egg-info` or `dist-info` or similar contain info about "extras"
which should be in their own subpackages like `%package -n
python3-%{pypi_name}+$FEATURE`.
For both cases:
* Summary/Description is %{summary} + some small comment
* %files is the %ghost for the files which generate dependency information
---
There are basically these cases where I think autosubpackages are needed:
* library - depending on SONAME, generate package (derived from SONAME) and the
devel part for it
* debuginfo - the whole debuginfo thing should be replaced by such mechanism
(so that it can be split from RPM)
* translations - somebody already came up with this few days ago to the IRC,
but in short we just need to generate separate subpackage (akin to
debuginfo-per-package) with translations
Things to not forget:
* Way how to inject some additional dependency or a file to subpackage (this
probably can be simply done by placing some files in %{buildroot} in some
specific places in some specific format
* Anything we do here should be possible to disable/override
---
I guess we need something like @ikeydoherty is describing about file pattern
matching (which we already have thanks to dependency generators), so probably
if we just extend that syntax to make them output some specially formatted
attributes (json? some subset of specs?) and then merge it.
So let's take case with some rust package which has a binaries, shared
libraries, devel stuff (with multiple features) and some custom utils subpkg.
That would mean that in buildroot there will be:
* `/usr/bin/foo`
* `/usr/lib64/libfoo.so` (SONAME is `libfoo.so.1`)
* `/usr/lib64/libfoo.so.1`
* `/usr/lib64/libfoo.so.1.0.0`
* `/usr/share/cargo/registry/foo-1.0.0/Cargo.toml` (with feature `a`)
The full spec would have something like (`auto:` prefix is what user should not
write, but would be auto-generated, omitting anything what dependency
generators would generate):
```
auto: # Any files which will not be used in other subpackages will go into the
main package
auto: %global _unmatched_files_in_main_package 1
…
Name: rust-foo
Summary: Something very useful
…
auto: %files
auto: %{_bindir}/foo
…
auto: %package -n libfoo-1
auto: Summary: %{summary} - libfoo.so.1
auto: %files -n libfoo-1
auto: %{_libdir}/libfoo.so.1
auto: %{_libdir}/libfoo.so.1.0.0
…
auto: %package -n rust-foo-devel
auto: Summary: %{summary} - Rust development files
auto: %files -n rust-foo-devel
auto: %{_datadir}/cargo/registry/foo-1.0.0/
…
auto: %package -n rust-foo+default-devel
auto: Summary: %{summary} - Rust development files for "default" feature
auto: %files -n rust-foo+default-devel
auto: %ghost %{_datadir}/cargo/registry/foo-1.0.0/Cargo.toml
…
auto: %package -n rust-foo+a-devel
auto: Summary: %{summary} - Rust development files for "a" feature
auto: %files -n rust-foo+a-devel
auto: %ghost %{_datadir}/cargo/registry/foo-1.0.0/Cargo.toml
…
%package doc
%files doc
%doc html
…
```
Now customization part is coming, I need to add %license into the
`rust-foo-devel` package, so:
```diff
-auto: %files -n rust-foo-devel
+%files -n rust-foo-devel
+%license LICENSE
```
should not throw error that package was not defined, but rather check it at the
end of build, after packages were generated.
Same if I decide to override summary of some subpackage, it should simply merge
them, keeping user-written changes with highest priority:
```diff
-auto: %package -n libfoo-1
-auto: Summary: %{summary} - libfoo.so.1
+%package -n libfoo-1
+Summary: Custom summary - libfoo.so.1
```
This means that for text fields we should override, but for arrays (like files)
we should append.
OTOH we probably should not allow such customizations in this way, but rather
have each generator output everything needed in lua or to the filesystem and
create special section `%subpackages` where there will be files like
`$pkgname.(summary|files)` generated by new generators. at that point it would
be very similar to the %generate_buildrequires section so that you can do
whatever you need with those files. I think this would be my preferred idea.
Thoughts?
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626306079___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
Forgot to mention that %subpackages section should store files in the
%{buildroot} too, so that there is possibility to write generators which would
depend on whole state of subpackages (current problem with dependency
generators).
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626306278___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
Oh yeah, this way we can solve problem described in #1073 but having some script which will put license thing into the `$pkgname.license`. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626306785___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
> So I guess this is waiting for me to put my thoughts here…
A lot of those things are already handled Fedora-side in our fonts and go
packaging macros.
1. you define a pivot `%{fooX}` variable, with X a suffixed index à la
%{SOURCEX}. If it is present in the spec file, that means you need to generate
the foo set of packages with foo rules (you can mix subpackages of different
types in a spec file, ie fonts + go in golang-x-image, you can have multiple
subpackage of the same type in the spec file, and a single control variable may
generate multiple kinds of subpackages, for example for Go we want to generate
go source module packages, gopath source packages, and eventually dynlib
packages. %{gomodX} means generating the three above subpackages for each
%{gomodX} line in the spec)
2. for each %{fooX} pivot variable, you define a set of optional sub-variables
that allow controlling the generation. For example %{gomodsummaryX} will
contain the summary of the Go module subpackage associated with %{gomodX}, and
if not set by the packager will be autocomputed to 'The %{gomodX} Go source
package'
You end up with a *huge* list of subvariables, that are only set in special
cases, so the average spec is kept small and maintainable.
Thus, we have a %{fontlicenseX} that allow setting the license of a font
subpackage (with a fallback to %{license}). Font file bundled distributed with
other stuff are usually subject to their own licensing. We have a
%{foolicensesX} that sets the licensing files associated with a subpackage. We
have a %{foodocsX} that sets the documentation files associated with a
subpackage
3. And lastly, we have %{fooheaderX} that lets the packager inject manual
header elements in generated subpackages, typically manual Provides/Obsoletes,
because that’s downstream stuff that can not be guessed from upstream metadata.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626312340___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
After lots of refactorings, I’re reduced the complexity of fonts/go (not published yer) header generation to the trivial https://pagure.io/fonts-rpm-macros/blob/009ccace3f337f3410cf0b4b789af692fce766d7/f/rpm/lua/srpm/fonts.lua#_135 And setting the rpm variables that uses in a safe way in presence of mutliple subpackages to https://pagure.io/fonts-rpm-macros/blob/009ccace3f337f3410cf0b4b789af692fce766d7/f/rpm/lua/srpm/fonts.lua#_118 The bulk of the complexity is identifying all the subvariables a domain will need, and defining the fallback rules between those variables (the thing that makes the "magic" work for users of the macros because most of the subvariables have sane default values that almost never need overriding). There’s a lot of domain knowledge at this stage, even some things like licensing files and documentation are needed for every domain. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626313382___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
And, you absolutely need the pivot and subvariables set spec-wide, in the
preamble or some early section, because a lot of the domain info will be used
in several spec sections, not just in %files, %build, %whatever.
For example Free Desktop people invented the idiotic appstream descriptor for
font files, which is so ass backwards even Gnome’s own Bistream Vera or
Cantarell ship it.
So one of the fonts subvariables if %{fontappstreamX}. If set, will use the
packager-specified appstream files (a couple font projects were sucked into
writing those before everyone noticed it was a waste of time). If not set font
package automation will compute those in `%install` from the files actually
installed at that point, and use the result in `%files` (would be cleaner to do
it as `%build` `%install` `%files` but at `%build` stage you do not know yet
which files will be installed and need appstreaming)
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626314630___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
> A lot of those things are already handled Fedora-side in our fonts and go > packaging macros. Sorry, I'm not interested in this black magic which nobody except you understand. I am interested in user-friendly solution which is supposed to be implemented in RPM. > You end up with a huge list of subvariables, that are only set in special > cases, so the average spec is kept small and maintainable. Exactly because of this. I don't want to have overcomplicated macros, I want simple configuration which I can tune to support different kinds of behaviors for different ecosystems. > https://pagure.io/fonts-rpm-macros/blob/009ccace3f337f3410cf0b4b789af692fce766d7/f/rpm/lua/srpm/fonts.lua#_135 I did not open a link, but I see that it points to line 135. I don't want to have anything in lua which is more than 10 lines. Better to not have lua at all involved here. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626315533___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
It is user friendly. It is not maintenance friendly because it workarounds rpm defficiencies (a lot of the complexity is creating lua arrays when rpm does expose an array element) > Exactly because of this. I don't want to have overcomplicated macros, I want > simple configuration which I can tune You can turn it all way round configuration means conf variables. The thing that scares people in fonts and go macro is reading and setting those variables using rpm primitives. > I did not open a link, Then perhaps you should do so before commenting on stuff you did not read -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626316072___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
@ignatenkobrain: Unlike the Go stuff, the fonts Lua macros are considerably simpler to understand, just there's a lot of functions. But @nim-nim, I agree that we need this functionality natively in RPM. The contortions that openSUSE goes through to generate flavor subpackages for Ruby and Python with their `%rubygem_subpackages`/`%python_subpackages` macros really indicate we need this functionality built-in to RPM. That does not obviate the need for macros to manage this behavior, but it does mean that we'd have better primitives for managing this. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626316771___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
> It is user friendly. It is not maintenance friendly because it workarounds > rpm deficiencies. A lot of the complexity is simulating arrays from > individual suffixed variables when rpm does expose an array element. That is exactly why I said having new section like `%subpackages` where anybody can do something like: ``` echo "MIT" > subpkg1.license sed -i -e "/^useless-thing.pdf$/d" libfoo.files ``` instead of dealing with arrays in lua or any other macros. No? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626316905___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)
> echo "MIT" > subpkg1.license
> sed -i -e "/^useless-thing.pdf$/
That’s actually much worse than what the go and fonts macro do. It’s only
simple because you’re thinking small with a single conf variable. And did not
code reading back, overriding and fallbacking those variables (common
configuration needs).
Multiply by the amount of variables correct domain generation needs, and that a
srpm can generate dozens of separate subpackages, and we’ll see who is writing
cryptic black magic code.
> @ignatenkobrain: Unlike the Go stuff, the fonts Lua macros are considerably
> simpler to understand, just there's a lot of functions.
It is simpler and cleaner because it’s a third-gen implementation of the
concept. The forge and go macros had to go first to help identify the generic
patterns needed for generation (and code all the small helpers that keep the
fonts code small and easy to read). The next set of go macros will start from
the factorisation achieved by the fonts macros (of course, go is intrinsically
harder, so it will probably stay more complex).
> But @nim-nim, I agree that we need this functionality natively in RPM.
I don’t disagree, that’s why I am commenting here with feedback after
implementing all this without rpm help Fedora-side for two different
application domains.
If you want autogeneration to work, the hard shell to crack is simple setting
and reading of the mass of configuration variables required to adjust automated
generation to real-world project quirks, on a subpackage level (each subpackage
may have its own quirks). It’s not the sole thing needed, but the other parts
are *easy* compared to this one.
Once you have good variable handling, generation becomes the trivial code I
posted
https://pagure.io/fonts-rpm-macros/blob/009ccace3f337f3410cf0b4b789af692fce766d7/f/rpm/lua/srpm/fonts.lua#_135
I did *not* begin three years ago with this kind of code. I tried to set and
read as few rpm variables as possible,and as late as possible JIT-way, because
it was so painful in current rpm. I tried to work with the rpm argument parser,
ended up with a wall of per-macro options that were never parsed as you wanted
them to, and options that needed tedious repeating in pretty much every section
of the spec
But, was forced by reality to bite the bullet, use global control variables
everywhere, set them early in the preamble (lastly, because of the rpm 4.15
Source: declaration clusterf*, but everything was already pointing towards
generic general global early setting of control variables), emulate arrays
because looping over subpackages in a spec requires looping over the same
variables for every subpackage.
The only reason the fonts macros are easy to read and maintain now is because
they benefit from a strong design framework where things like
%{currentfontlicense} just exist at the point the automation logic needs it to
exist.
--
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/329#issuecomment-626320398___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint
Re: [Rpm-maint] [rpm-software-management/rpm] RFC: Removing unneeded internal macros (such as %__ranlib) (#1211)
Rather than go mass destruct all at once, it's better to try find things that are obviously entirely unused by everybody, whether through plain obsolescence (such as those long obsoleted %_foo*bin compat macros) or the fact that they cannot be meaningfully used within rpm context. And keep on nipping a little by little. For example there are requests to set CC from %__cc (and the related stuff) in %configure, I wouldn't be so quick to dismiss those as although %__cc is unused in rpm, we do deal with compiler flags quite a bit. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/1211#issuecomment-626488312___ Rpm-maint mailing list Rpm-maint@lists.rpm.org http://lists.rpm.org/mailman/listinfo/rpm-maint
