[Rpm-ecosystem] New RPM community venue

[Panu Matilainen]

As of today, we're opening up the GitHub Discussions forum as a new 
venue for community interaction:


https://github.com/rpm-software-management/rpm/discussions

Why, you ask, when we have all these mailing lists? The sad fact is that 
the mailing lists are all but dead, to the point that even us 
maintainers miss the rare post on them, leading them even more dead 
because few people like talking to themselves. Yet, clearly there is a 
need for a place to ask questions and discuss various aspects of rpm and 
its future, and based on evidence people are more inclined to file a 
ticket to do this rather than post on a mailing list. That, or remain 
silent. Neither is a particularly good outcome.


We hate the potential vendor lock-in as much as anybody, so these 
discussions will always be mirrored to rpm-maint mailing list along with 
the ticket and PR notifications. Other than that, we'll see how it goes.


On behalf of rpm-team,

- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] rpm logs to stdout/stderr when callback is set

[Panu Matilainen]

On 3/25/22 14:47, Lukáš Hrázký wrote:

On Tue, 2022-03-15 at 15:48 +0100, Lukáš Hrázký wrote:

Hello everyone,

I'm setting up rpm logging in libdnf 5 and even after setting a log
callback function via rpmlogSetCallback(), rpm duplicates all messages
to stdout/stderr.

This is extremely impractical, can something be added to the API so
that I can turn it off?

Ideally rpm shouldn't log to stdout/stderr if a callback is set and I'd
be in favor of changing this, but I'm not sure you'll want to do that.


After talking with Panu turns out this can be controlled via the log
callback return value, so not an issue.

Shifting the topic but I thought I'd keep the discussion in one thread,
a couple other issues of varying severity / implications I've bumped
into:

(1) There are instances when the code writes to stdout/stderr directly
instead via logging:

One case at this spot:
https://github.com/rpm-software-management/rpm/blob/a6913834d395d6544c2ba1578d6ebd594350b602/rpmio/rpmio.c#L1448


Right. That particular case should only really print out anything when 
rpmio_debug is enabled, those stats really are not that interesting in 
this day and age.




Another one I haven't managed to find in the code yet but it logs this:
Header SHA256 digest: OK
Header SHA1 digest: OK


These come from 
blob/a6913834d395d6544c2ba1578d6ebd594350b602/rpmio/rpmio.c#L1448 - they 
are rpmlog()'ed so it's something else. Maybe they just happen before 
you have the callback set up, eg keyring load would log these for any 
gpg-pubkeys found.



I think these are easy to fix and I'd like to get them fixed, as they'd
interleave with our normal stdout/stderr output (in case DEBUG log mask
is enabled).


Please report as GH ticket(s), email somewhere on a mostly forgotten 
mailing list is too easy to forget :D



There may be more of these but my cursory grepping of the repo hasn't
found anything.


(2a) The log callback is stored in a static variable, meaning rpm can't
be used from two different places of the same process simultaneously.
In dnf we have a context-style Base object and nothing is stopping the
user to create multiple Bases with different configurations. And then
potentially install via rpm into different installroots simultaneously.

I don't think there's really a reasonable solution to this, I'm just
bringing it up for awareness purposes.


You couldn't install to different installroots anyway because chroot() 
is process global. So while its (supposedly) okay to have multiple 
transaction sets on different databases, rpmtsRun() will always need to 
be serialized for very concrete reasons. Of course this isn't actually 
enforced in the code, most of librpm simply isn't thread-safe, and even 
the parts that are require treading carefully.


Supporting multiple log contexts wouldn't be too hard as we already have 
all the relevant stuff in one struct. It'd be mostly just adding 
context-aware API variants alongside the current ones, plus some means 
of setting eg the per-ts logger to use, a lot of churn to update all the 
relevant rpmlog() calls to actually use it. Feel free to file a ticket 
if this is something that seems important to dnf.



(2b) Somewhat related, due to the above limitation I'm setting the rpm
log callback only just before we start working with an rpm transaction
and lock this with a static variable lock so that the log messages
don't get misdirected. The API returns the previous log callback so it
could be reset after the lock is released, but the API doesn't provide
a way to access the data of the log callback, so restoring the callback
would just break horribly when there's a different sort of data than
what it expects.

I don't think we necessarily need to restore the callback to its
previous value in dnf 5 so again just bringing this up for awareness.
This could be fixed by adding some API.


Yup, this is a flaw shared by more than one callback APIs in rpm. Please 
file a ticket so it gets tracked.


- Panu -



Here's the PR for integrating rpm logging into dnf 5:
https://github.com/rpm-software-management/libdnf/pull/1475


And here's a sample of the log, it's an excerpt from our unittests
output, you can see the direct stdout/stderr messages randomly
interleaving:
https://pastebin.com/nGn1b9gL

Cheers,
Lukas

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem


___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] RFC: Relocate RPM and DNF databases to /usr

[Panu Matilainen]

On 12/15/21 23:08, Chris Murphy wrote:

On Wed, Dec 15, 2021 at 4:09 AM Panu Matilainen  wrote:


On 12/14/21 22:59, Colin Walters wrote:



On Thu, Dec 9, 2021, at 10:11 AM, Chris Murphy wrote:


The change is not so simple. It is not only the movement of files from one 
location to another one. We store more types of data in that location - history 
database (sqlite), module failsafe data (yamls). In future we will store system 
state data (toml). Data is not only modified after RPM transactions but also by 
module and mark commands. What I want to say is that the change will be painful 
but in the proposal there are limited benefits.


When the rpmdb was bdb and not extensible, I understand the idea of dnf having 
its own separate database.  But I don't understand why there are two separate 
sqlite databases now.


The rpmdb is not hard-wired to sqlite. Also, whatever the format used
it's strictly private to rpm except what's exposed through librpm APIs.
So it's certainly NOT extensible in the sense that anybody wanting to
put something package related there is free to do so.



Anyways, a lot going on here and rather than point by point I'll say basically:

It'd be great from the rpm-ostree PoV to have a shared consistent place for the 
rpmdb.  Last time this came up I think Panu really wanted to get the sqlite 
change out first, which - fair.  But now that's done, so are there any other 
blockers?


Just the pain and misery of dealing with a fundamental change that has
zero benefit to rpm itself. It'll be a long, long, long time before
/var/lib/rpm is phased out of existence for good, and in the meanwhile
us rpm f get to answer all the questions and complaints about it. I'm
not looking forward to that.


The alternative to the proposal to switch rpmdb from /var to /usr, is
keeping it where it is. And provide matching between /usr and
/var/lib/rpm generations (or versions) by separate subvolume on Btrfs,
e.g. "varlibrpm" mounted at /var/lib/rpm. The separation is very cheap
on Btrfs, perhaps 200 KiB, because it's just a file btree. The cost is
a little more than a directory. And the snapshots only consume space
representing the difference between the snapshots.

However, for other implementations, such as  and separate file system
volumes for LVM+ext4 LVM+XFS or Stratis, it's not as cheap. Each
snapshot includes the file system journal, which varies in size.
Storage cost could be anywhere from 8 - 800 MiB per snapshot.

I'm not sure if the alternative is that great either, considering
movement is in progress to put rpmdb in /usr already.


My grumbles aren't so much about rpmdb in /usr as a concept, it's the 
move of a very very long standing "well known location". Oh, Colin 
actually asked for blockers and I didn't mean this is a blocker, just an 
unpleasantness left to deal with. Sorry about that.


It'd minimize the pain somewhat if the db location was left at 
/var/lib/rpm on rpm level but /var/lib/rpm being a symlink pointing 
inside /usr until all active Fedora versions had this, and at that point 
changing the rpm configuration would make no difference to anybody 
(within Fedora). But such a construct almost certainly wont survive 
rpmdb rebuilds so it's probably not practical.


- Panu -




--
Chris Murphy



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] RFC: Relocate RPM and DNF databases to /usr

[Panu Matilainen]

On 12/14/21 22:59, Colin Walters wrote:



On Thu, Dec 9, 2021, at 10:11 AM, Chris Murphy wrote:


The change is not so simple. It is not only the movement of files from one 
location to another one. We store more types of data in that location - history 
database (sqlite), module failsafe data (yamls). In future we will store system 
state data (toml). Data is not only modified after RPM transactions but also by 
module and mark commands. What I want to say is that the change will be painful 
but in the proposal there are limited benefits.


When the rpmdb was bdb and not extensible, I understand the idea of dnf having 
its own separate database.  But I don't understand why there are two separate 
sqlite databases now.


The rpmdb is not hard-wired to sqlite. Also, whatever the format used 
it's strictly private to rpm except what's exposed through librpm APIs. 
So it's certainly NOT extensible in the sense that anybody wanting to 
put something package related there is free to do so.




Anyways, a lot going on here and rather than point by point I'll say basically:

It'd be great from the rpm-ostree PoV to have a shared consistent place for the 
rpmdb.  Last time this came up I think Panu really wanted to get the sqlite 
change out first, which - fair.  But now that's done, so are there any other 
blockers?


Just the pain and misery of dealing with a fundamental change that has 
zero benefit to rpm itself. It'll be a long, long, long time before 
/var/lib/rpm is phased out of existence for good, and in the meanwhile 
us rpm f get to answer all the questions and complaints about it. I'm 
not looking forward to that.


- Panu -



We could try simply not changing existing systems in-place at first, but just 
do it for new installs, add the backcompat symlink, ensure the stack looks in 
both places (preferring new); that's actually mostly done AFAIK.

Today rpm-ostree has its own state management that doesn't use the libdnf 
history stuff.  Longer term it would be good to have a unified vision there - 
most of this is touched on in https://github.com/coreos/rpm-ostree/issues/2326

But I think I'm arguing to decouple the rpmdb move from the dnf move.



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] debugedit project setup

[Panu Matilainen]

On 3/23/21 2:29 PM, Panu Matilainen wrote:

On 3/22/21 7:42 PM, Mark Wielaard wrote:

Hi,

As discussed I setup a separate debugedit project:
https://sourceware.org/debugedit/

It contains the debugedit and sepdebugcrcfix programs and the find-
debuginfo.sh script and the debugedit.at testsuite. I used git-filter-
repo to keep the git history of the files from the rpm repo. Then I
resynced the libiberty and binutils files and added a minimal
automake/autoconf/autotest build system. Replace popt with getopt
argument parsing. And use libiberty md5 or sha1 instead of rpmDigest
algorithms for build-id updates.

There is a test debugedit-0.1 release to see whether this works. But I
would like to make sure first that things are setup so that flatpak and
rpm can use this as is before people start packaging it. Once we know
it can be used as replacement for the built-in rpm/flatpak debugedit we
do a proper debugedit 1.0 release.


Awesome. I'll make it a priority to see that we transition, if at all 
possible, to the external debugedit in rpm 4.17 already. Stay tuned...


Here goes:

https://github.com/rpm-software-management/rpm/pull/1712

- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Using librpm directly to parse RPM databases

[Panu Matilainen]

On 3/25/21 5:28 PM, Richard W.M. Jones wrote:

Libguestfs is a library for inspecting disk images.  It has long
offered a way to find out what packages are installed in a Linux
virtual machine by inspection of the disk image.  For guests
identified as RPM-based it currently does this by some hokey parsing
of /var/lib/rpm/Name etc.  Since RPM moved away from BDB to sqlite
this has obviously stopped working, and now I'm trying to fix this.

An important thing here is that we still need to be able to parse out
the RPM databases from very old guests (eg. back to RHEL 5 / 2007 era
would be a good starting point).

The main operations of interest are:

  - List all installed RPMs.

  - Get some of the standard fields of an installed RPM, eg. name,
version, arch, URL, summary, description.

Based on playing with the programs here:
https://blog.fpmurphy.com/2011/08/programmatically-retrieve-rpm-package-details.html
I think we can probably do this using librpm directly.  My questions
below are about this option, but if using librpm directly is _not_
advisable for some reason then that would be good to know too.

* How stable is the librpm API?  If we switched to this method then
we'd probably be using it for years.  The programs above date from
2011 and still compile and work even with sqlite, which is
encouraging.


The main parts of librpm API are rather stable, the relevant APIs for 
this sort of use hasn't really changed since 2008 (rpm 4.6.0, that's a 
kind of a watershed release).


Note that the examples in the above link are doing things the hard way - 
if all you want from rpmdb is formatted strings then you'll want to use 
headerFormat() (headerSprintf() prior to 4.6) which is the API 
equivalent of rpm --queryformat and far nicer and simpler for that purpose.




* Will downstream versions of librpm maintain the ability to at least
read BDB databases forever?  Not interested in writing.


Forever is such a long time :D

Upstream will need to support reading BDB databases as long as there are 
supported releases using BDB in the wild. For a ballpark figure, I would 
say the read-only support will remain at least for the lifetime of 
RHEL-8 and then some. Which puts us somewhere beyond 2030 - too far to 
further predictions.


Downstream support is a distro matter, the bdb-ro support is strictly 
optional in rpm build. As for Fedora, I'd estimate the same as upstream, 
but I can't speak for others.



* Are there security implications to reading an RPM database, ie.  if
the database has been corrupted, perhaps deliberately, do we need to
confine or time-box the librpm process?  (We propose to confine it to
a VM so this is more about DoS attacks.)


I don't know about security implications, but confining librpm access to 
a process of it's own is not a bad idea at all, even if only to protect 
protect *yourself* from the librpm side-effects such as signal delivery 
hijack when rpmdb is open. The other thing is that with native BDB, 
access by a process with write permission to the db directory can have 
side-effects to the database environment. To guard against that, you 
might want to run it as a some sort of nobody-user.



A questions about the API:

* The example programs call rpmReadConfigFiles().  I wonder if we
should _not_ do that because of security or other considerations?


Most of rpm will simply not work at all without calling 
rpmReadConfigFiles(), so it's not really optional.


Note that I'm quite sure I don't fully understand the use-case with 
libguestfs. If you can link to librpm and isolate it to its own process, 
couldn't you just exec the actual rpm binary for queries? The command 
line query interface is essentially compatible till the beginning of times.


- Panu -



Rich.



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] debugedit project setup

[Panu Matilainen]

On 3/23/21 2:29 PM, Panu Matilainen wrote:

On 3/22/21 7:42 PM, Mark Wielaard wrote:

Hi,

As discussed I setup a separate debugedit project:
https://sourceware.org/debugedit/

It contains the debugedit and sepdebugcrcfix programs and the find-
debuginfo.sh script and the debugedit.at testsuite. I used git-filter-
repo to keep the git history of the files from the rpm repo. Then I
resynced the libiberty and binutils files and added a minimal
automake/autoconf/autotest build system. Replace popt with getopt
argument parsing. And use libiberty md5 or sha1 instead of rpmDigest
algorithms for build-id updates.

There is a test debugedit-0.1 release to see whether this works. But I
would like to make sure first that things are setup so that flatpak and
rpm can use this as is before people start packaging it. Once we know
it can be used as replacement for the built-in rpm/flatpak debugedit we
do a proper debugedit 1.0 release.


Awesome. I'll make it a priority to see that we transition, if at all 
possible, to the external debugedit in rpm 4.17 already. Stay tuned...


FWIW, here's a quick-n-dirty package of debugedit 0.1:

https://laiskiainen.org/rpm/debugedit/

- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] debugedit project setup

[Panu Matilainen]

On 3/22/21 7:42 PM, Mark Wielaard wrote:

Hi,

As discussed I setup a separate debugedit project:
https://sourceware.org/debugedit/

It contains the debugedit and sepdebugcrcfix programs and the find-
debuginfo.sh script and the debugedit.at testsuite. I used git-filter-
repo to keep the git history of the files from the rpm repo. Then I
resynced the libiberty and binutils files and added a minimal
automake/autoconf/autotest build system. Replace popt with getopt
argument parsing. And use libiberty md5 or sha1 instead of rpmDigest
algorithms for build-id updates.

There is a test debugedit-0.1 release to see whether this works. But I
would like to make sure first that things are setup so that flatpak and
rpm can use this as is before people start packaging it. Once we know
it can be used as replacement for the built-in rpm/flatpak debugedit we
do a proper debugedit 1.0 release.


Awesome. I'll make it a priority to see that we transition, if at all 
possible, to the external debugedit in rpm 4.17 already. Stay tuned...




Please join the debugedit mailinglist for discussions:
https://sourceware.org/mailman/listinfo/debugedit

File bugs or feature requests:
https://sourceware.org/bugzilla/enter_bug.cgi?product=debugedit

Checkout the git repo:
git clone git://sourceware.org/git/debugedit.git

I have played a bit with pagure, but haven't set it up properly, sorry.
So for now we'll do patches by email.


I for one far prefer patches by email over Pagure...

- Panu -



Cheers,

Mark
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] rpm debugedit as a separate project?

[Panu Matilainen]

On 2/19/21 10:23 PM, Mark Wielaard wrote:

Hi all,

rpm debugedit has grown from a quick hack that simply listed/replaced
some files/strings to an almost full blown DWARF reader/writer. It is
now also used outside rpm(build). Debian packages it separately and
Flatpak builder has an embedded copy it uses to post-process debuginfo.

It is currently not always easy to update because the people who
contribute to debugedit are often not regular rpm contributors. And the
release cycle of rpm doesn't always match up with the release cycle of
the GNU toolchain. So it sometimes needs a release at a different time
than rpm gets released.

There have been some requests to have it moved from rpm to elfutils:
https://sourceware.org/bugzilla/show_bug.cgi?id=27351
But I think it would be simpler to have it be its own project, so it
isn't tied to another projects release schedule and processes.


As already stated elsewhere, the rpm developers fully support and 
encourage this, regardless of the exact whereabouts. If it makes more 
sense as a separate project to you, then that's absolutely fine.




So my proposal is to:

- Create a debugedit project on sourceware, so it is close to binutils,
   from which it sometimes steals code, elfutils on which it relies for
   libelf/libdw, and dwz which is a similar, but completely different
   DWARF processor. Most people currently contributing to rpm debugedit
   should already have an account there.

- Import tools/debugedit.c tools/hashtab.c tools/hashtab.h and
   tests/debugedit.at from rpm. Add a minimal build using autoconf and
   autotest around this. Update the hashtab files from libiberty,
   check debugedit (and sepdebugcrc checkm see below) for updates
   which came in from binutils. Note, it also has a popt dependency.

- Setup buildbot using builder.wildebeest.org/buildbot which has
   support for debian/fedora/centos, armhf, i386, x86_64, aarch64,
   s390x, ppc64 and ppc64le.

- Provide patches for rpm to have some kind of --with-system-debugedit
   configure flag so it won't build and install its own debugedit
   but picks up an installed debugedit on the system.

- Provide patches for flatpak-builder to use debugedit like it already
   uses eu_elfcompress and eu_strip, instead of calling
   builder_get_debuginfo_file_references.

- Setup the buildbot so it runs the rpm and flatpak-builder testsuites
   against debugedit to make sure we keep compatibility.

   This should in theory be easy because both have testsuites that
   should be zero-fail. But in practice I never got the flatpak-builder
   tests all green because I don't understand what it is doing with
   gpg-agent. And I always trip over the usage of fakechroot in the rpm
   testsuite, on some setups it seems fakechroot is just totally broken?

An open question is for how long rpm and flatpak-builder want to keep
their internal implementation. And if so, how we keep them in sync. Or
if we simply decide that once debugedit is ready as separate project
the next release of rpm and flatpak-builder will simply require
debugedit as external executable.


As for rpm, we'll simply require an external debugedit from the go, ie 
as soon as there is a released standalone debugedit exists.




Another question is whether the separate debugedit project should also
adopt some of the other related tools like sepdebugcrcfix, elfdeps and
maybe scripts/find-debuginfo.sh? sepdebugcrcfix and elfdeps seem easy
to adopt. find-debuginfo.sh is very tightly linked to how rpm builds
debuginfo/sources subpackages. But maybe it could be made a little bit
more generic? But if so, keeping compatibility might be tricky.


sepdebugcrcfix makes even less sense in rpm if debugedit goes, so yeah 
please adopt that too. find-debuginfo.sh might be tricky, but something 
I would prefer to see moved too - eventually if not at the start.


As for elfdeps, unlike debugedit and sepdebugcrc it's just a simple 
libelf client which walks ELF structures and maps it to fairly 
rpm-specific output (those wacko "()(64bit)" markers only on select 
64bit arches and all) , and has not been any sort of maintenance burden 
from my POV. I see no need to move it anywhere as it is.




I don't think a separate debugedit project needs much maintenance once
setup. But there are a couple of items to work on. In particular
support for DWARF5 as emitted by alternative compilers and handling
Split Dwarf.

Comments and feedback more than welcome.


I have mostly just one: thank you for this initiative!

- Panu -



Cheers,

Mark
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Using RPM with NDB on macOS

[Panu Matilainen]

On 10/21/19 12:43 PM, Panu Matilainen wrote:

On 10/20/19 3:47 AM, Neal Gompa wrote:

My current plan is to at least get to the point where I can try
working with the solv(1) demo program included in libsolv, and then
proceed from there. At least with the DNF stack, there's still the
problem that it assumes bdb[6] and specific rpmdb paths. I'd like to
get to the point that I could use MicroDNF[7]...

[1]: 
https://github.com/rpm-software-management/rpm/pull/899#issuecomment-543129651 


[2]: https://github.com/rpm-software-management/rpm/issues/334
[3]: 
https://github.com/rpm-software-management/rpm/commit/3625ca14c752fa229c79891fcc6374df40b5b588 


[4]: https://github.com/rpm-software-management/rpm/pull/907
[5]: 
https://github.com/rpm-software-management/rpm/compare/rpm-4.15.x...Conan-Kudo:rpm-4.15.0-osx-ndb 


[6]: https://github.com/rpm-software-management/libdnf/issues/362
[7]: https://github.com/rpm-software-management/microdnf



Add https://github.com/openSUSE/libsolv/issues/354 to the list, it's 
essentially the very same thing as libdnf ticket #362.


I've dnf running on top of sqlite rpmdb now. For sqlite this is rpm from 
git master, but for other backends, released rpm versions can be used.


libsolv upstream has the necessary bits, I'm testing on 0.7.7 for which 
you'll need to have these:
- 
https://github.com/openSUSE/libsolv/commit/94df5d6bf2eec3fc50307342e250e2b842c6
- 
https://github.com/openSUSE/libsolv/commit/ea037722b0f2a1afc8416e08b36fd498a8bb14c4


In addition, libdnf minimally needs this (the rpmdb caching wont be 
functional but it'll at least work), tested against libdnf-0.35.5:

https://github.com/rpm-software-management/libdnf/pull/824

- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Using RPM with NDB on macOS

[Panu Matilainen]

On 10/20/19 3:47 AM, Neal Gompa wrote:

Hey all,

As I had alluded to in the pull request adding the SQLite rpmdb
backend[1], I have been doing some work every once in a while for the
past few years on rpm for macOS.

A point of frustration has been having a working bdb package, and
maintaining bdb is functionally impossible for me, as I have no
capability to fix bugs there.

I had been experimenting with LMDB since it was introduced.
Unfortunately, it turned out to be more problematic than I expected.
Originally, NDB didn't work on macOS because of its usage of mremap()
(which does not exist on macOS)[2]. Thankfully, Michael Schroeder
recently fixed this in git master[3], so I gave it another try.

After a minor issue[4] and a backport of the ndb fixes to rpm 4.15[5],
I was able to transition to ndb and start trying it.

I've gotten rpm itself working with ndb, and it seems like all the
basic actions work perfectly fine with ndb. I'm working on building up
a package manager stack so that I can start doing some more
interesting tests, but there are issues with libsolv where it makes
the assumption about where the rpmdb path is and what type of rpmdb it
is...

My current plan is to at least get to the point where I can try
working with the solv(1) demo program included in libsolv, and then
proceed from there. At least with the DNF stack, there's still the
problem that it assumes bdb[6] and specific rpmdb paths. I'd like to
get to the point that I could use MicroDNF[7]...

[1]: 
https://github.com/rpm-software-management/rpm/pull/899#issuecomment-543129651
[2]: https://github.com/rpm-software-management/rpm/issues/334
[3]: 
https://github.com/rpm-software-management/rpm/commit/3625ca14c752fa229c79891fcc6374df40b5b588
[4]: https://github.com/rpm-software-management/rpm/pull/907
[5]: 
https://github.com/rpm-software-management/rpm/compare/rpm-4.15.x...Conan-Kudo:rpm-4.15.0-osx-ndb
[6]: https://github.com/rpm-software-management/libdnf/issues/362
[7]: https://github.com/rpm-software-management/microdnf



Add https://github.com/openSUSE/libsolv/issues/354 to the list, it's 
essentially the very same thing as libdnf ticket #362.


- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Source RPM provides?

[Panu Matilainen]

On 3/12/19 4:16 PM, Pat Riehecky wrote:
I've been doing a fair bit of mapping back and forth from binary rpms to 
source rpms.  And I had a question:


Would it make sense for a source rpm to have a generated 'Provides:' for 
any %package defined within the spec?


For example (actual syntax to be determined by folks smarter than myself):

$ rpm -qp --provides man-db-2.8.4-4.fc30.src.rpm
rpmbuild(man-db)
rpmbuild(man-db-cron)
$

I'd expect it to 'provide' all the %packages (even ones masked for your 
specific arch so that we don't deal with executing macros at query and 
whatnot).


Thoughts?


Seems like a perfectly reasonable request to me. Took the liberty of 
creating an RFE ticket out of it:


https://github.com/rpm-software-management/rpm/issues/642

- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Source RPM provides?

[Panu Matilainen]

On 3/12/19 5:00 PM, Neal Gompa wrote:

On Tue, Mar 12, 2019 at 10:49 AM Pat Riehecky  wrote:


I've been doing a fair bit of mapping back and forth from binary rpms to
source rpms.  And I had a question:

Would it make sense for a source rpm to have a generated 'Provides:' for
any %package defined within the spec?

For example (actual syntax to be determined by folks smarter than myself):

$ rpm -qp --provides man-db-2.8.4-4.fc30.src.rpm
rpmbuild(man-db)
rpmbuild(man-db-cron)
$

I'd expect it to 'provide' all the %packages (even ones masked for your
specific arch so that we don't deal with executing macros at query and
whatnot).

Thoughts?



It'd probably be better if we had an equivalent to debbuild's
--showpkgs switch that parsed the spec file and identified what
packages would be built from it.


You mean like this?

[pmatilai︎sopuli rpm]$ rpmspec -q --builtrpms rpm.spec
rpm-4.14.2.1-4.fc29.1.x86_64
rpm-libs-4.14.2.1-4.fc29.1.x86_64
rpm-build-libs-4.14.2.1-4.fc29.1.x86_64
rpm-sign-libs-4.14.2.1-4.fc29.1.x86_64
rpm-devel-4.14.2.1-4.fc29.1.x86_64
rpm-build-4.14.2.1-4.fc29.1.x86_64
rpm-sign-4.14.2.1-4.fc29.1.x86_64
python2-rpm-4.14.2.1-4.fc29.1.x86_64
python3-rpm-4.14.2.1-4.fc29.1.x86_64
rpm-apidocs-4.14.2.1-4.fc29.1.noarch
rpm-cron-4.14.2.1-4.fc29.1.noarch
rpm-plugin-selinux-4.14.2.1-4.fc29.1.x86_64
rpm-plugin-syslog-4.14.2.1-4.fc29.1.x86_64
rpm-plugin-systemd-inhibit-4.14.2.1-4.fc29.1.x86_64
rpm-plugin-ima-4.14.2.1-4.fc29.1.x86_64
rpm-plugin-prioreset-4.14.2.1-4.fc29.1.x86_64
rpm-debuginfo-4.14.2.1-4.fc29.1.x86_64
rpm-debugsource-4.14.2.1-4.fc29.1.x86_64

- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Required version of rpm?

[Panu Matilainen]

On 06/26/2018 04:46 PM, Miroslav Suchý wrote:

Dne 26.6.2018 v 14:57 Panu Matilainen napsal(a):

On 06/26/2018 02:15 PM, Miroslav Suchý wrote:

Dne 26.6.2018 v 12:17 Neal Gompa napsal(a):

rpmlib() dependencies are virtual, they aren't provided by anything,
but are processed during the transaction and verified.


1) So the number in rpmlib(RichDependencies) means what version of rpm I should 
have. Right?


Roughly yes, but beware there are cases where this is not true at all due to 
backports and sideports messing it all up.


OK - corner cases and backports aside. But... I am still puzzled.

$ rpm -qR redhat-rpm-config |grep Rich
rpmlib(RichDependencies) <= 4.12.0-1

So any rpm with verision *lower* than 4.12.0 will work? Should be there ">=" ?


Nope. From my previous email: "the version in rpmlib() dependencies is 
sorta backwards to what people generally expect." - but it's the range, 
not the version that appears backwards.


This is the way all rpmlib() dependencies have always been, here's the 
best explanation I've found:


https://www.redhat.com/archives/rpm-list/2001-April/msg00283.html

And in fact the rich dependency tracking version is a fine example of 
how the version is not trustworthy: rich dependencies got added in 
4.13.0, not 4.12.0 as the rpmlib() dependency would have you think.



Not that you'd want to go telling to upgrade their rpm beyond what their distro 
provides, that's a recipe for disaster.


No, I do not want to do that. I just want to provide that user-friendly error.


Ack, good.

- Panu -



Miroslav
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Required version of rpm?

[Panu Matilainen]

On 06/26/2018 02:15 PM, Miroslav Suchý wrote:

Dne 26.6.2018 v 12:17 Neal Gompa napsal(a):

rpmlib() dependencies are virtual, they aren't provided by anything,
but are processed during the transaction and verified.


1) So the number in rpmlib(RichDependencies) means what version of rpm I should 
have. Right?


Roughly yes, but beware there are cases where this is not true at all 
due to backports and sideports messing it all up.


What you really need is to have an rpm version that provides the virtual 
capability in question (discoverable via --showrc and API), the version 
in rpmlib() dependencies is sorta backwards to what people generally expect.



2) The version of rpm I need to parse this RPM is in dependency list, which I 
cannot parse on EL7 because there is too
old rpm. So I have chicken and egg problem. Any idea how to solve this?


Yeah rich dependencies complicated the scene quite a bit. What was 
previously a dumb string no longer is, and could fail to be parsed by 
rpmds. What you always can get though, is the "raw" strings from the 
header, in RPMTAG_REQUIRENAME, RPMTAG_REQUIREFLAGS and 
RPMTAG_REQUIREVERSION arrays and look for rpmlib( in start of the name 
(it's supposed to be also flagged in RPMTAG_REQUIREFLAG but older rpm 
versions didn't always add it).


Not that you'd want to go telling to upgrade their rpm beyond what their 
distro provides, that's a recipe for disaster. But sure, being able to 
detect such issues and provide a nicer explanation never hurt anybody.


- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Two signatures?

[Panu Matilainen]

On 05/31/2018 04:40 PM, Miroslav Suchý wrote:

In past, there was possible to add two or more signatures to rpm package. At 
least according to
   http://ftp.rpm.org/max-rpm/s1-rpm-pgp-signing-packages.html

But when I checked current rpmsign, it seems that --addsign actually replace 
the previous signature.
Is this correct? Or is there way to have two signatures on one package?



Short version:
As of all rpm versions in the last 15+ years, --addsign/--resign do the 
same thing which is replace any existing signature, so no, you cannot 
pile them on.


Longer version:
Technically it's more complicated than that - there are always two 
different signatures (one on the header and one on header + payload) but 
always by same signer and algorithm for both. And technically, the 
header could store more signatures, in particular DSA and RSA 
simultaneously, it's just capped in the code for simplicitys sake. 
Further, there are provisions for true multiple signature support in the 
codebase in rpm >= 4.14.x but the actual support for that hasn't landed 
yet. Maybe later this year.


- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] possible signature types

[Panu Matilainen]

On 05/09/2018 11:27 AM, Miroslav Suchý wrote:

Hi,
I am working on updating documentation about signing a packages using rpmsign. 
In the old documentation there is:


The "Generating signature" message appears in both the binary and source 
packaging sections. The number following the message indicates that the signature added 
was created using PGP.

[NOTE]
--
The list of possible signature types can be found in the RPM sources, 
specifically signature.h in RPM's lib subdirectory.
--




The problem is that I see nothing relevant in current signature.h.
So what is the recommended way to get list of possible signatures nowadays?


The comment about signature.h refers to something so old you don't want 
know, much less care.


There's only one type of signature that rpm currently supports: OpenPGP.

Note that in rpm internal jargon, header/package level hashes are 
sometimes referred to as signatures too due to them residing in the so 
called signature header. Which can be confusing even if you know it.


- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Trying to understand %buildsubdir and debuginfo generation

[Panu Matilainen]

On 04/27/2018 06:47 AM, Jason L Tibbitts III wrote:

"JJ" == Jeff Johnson  writes:


JJ> Note that the macro definitions you are trying to understand are not
JJ> from rpm itself (so you perhaps should be consulting with other
JJ> SME's than me).

Well, I was consulting a mailing list.

Here's a paste from the macros.debug file in the rpm 4.14.1 source:

%install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\
%%install\
%{nil}


Note that macros.debug is only used inside the rpm testsuite, none of it 
is active in normal installation.




Here's a paste from the macros that Fedora uses
(/usr/lib/rpm/redhat/macros):

%install %{?_enable_debug_packages:%{?buildsubdir:%{debug_package}}}\
%%install\
%{nil}


Ew. Now there's a blast from the past, that ugly %install override is 
from 2002 when debuginfo was a fairly new concept. If that's actually 
needed for debuginfo then it's a bug (in rpm).




Seems... similar.

JJ> Talk to whomever decided that there was a need to overload %install
JJ> with a test on %buildsubdir.

OK, so, again, I'm trying to talk to this mailing list, which I thought
provided the best chance to reach the people who know.


Well, git blame brings this up (redhat-rpm-config):

commit 1640cd5cac6e1308de8f303e95089053e7f5c9b5 (tag: 
REDHAT_RPM_CONFIG_8_0_14)

Author: Tim Powers 
Date:   Sat Dec 14 20:37:32 2002 +

more debug macro tweaks

...and that's all he wrote. Gotta love the descriptive commit messages 
of that era.


- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Fwd: [Rpm-maint] Fixing macro scoping

[Panu Matilainen]

On 02/06/2017 05:00 PM, Panu Matilainen wrote:

On 02/06/2017 04:12 PM, Neal Gompa wrote:

On Mon, Feb 6, 2017 at 4:50 AM, Panu Matilainen
<pmati...@laiskiainen.org> wrote:

On 02/06/2017 10:56 AM, Vít Ondruch wrote:




Dne 3.2.2017 v 13:14 Panu Matilainen napsal(a):



Stunned silence on rpm-maint, forwarding to rpm-ecosystem in hopes of
a larger and livelier audience...

- Panu -

 Forwarded Message 
Subject: [Rpm-maint] Fixing macro scoping
Date: Mon, 23 Jan 2017 12:30:21 +0200
From: Panu Matilainen <pmati...@laiskiainen.org>
To: rpm-ma...@lists.rpm.org <rpm-ma...@lists.rpm.org>


Consider the following snippet, originating from
https://bugzilla.redhat.com/show_bug.cgi?id=552944:

%{!?foo: %define foo bar}
%define dofoo() true
echo 1: %{foo}
%dofoo
echo 2: %{foo}

I'd assume everybody agrees both %{foo}'s should evaluate to the same
value, but that is not the case currently. Using a cli-variant of the
above:

[pmatilai@sopuli rpm]$ rpm --define 'dofoo() true' --eval '%{!?foo:
%define foo bar}' --eval '%foo' --eval '%dofoo' --eval '%foo'
warning: Macro %foo defined but not used within scope

bar
true
%foo

The flaw here is that rpm supposedly implements block level scoping
for macros (so in the above example, "foo" would only exist in the
{!?foo:...} block), but doesn't actually enforce that, unless a
parametric macro is "called". Current rpm warns about it, but warnings
or not this behavior doesn't make the slightest sense.

The question is, what do you think %{foo} should evaluate to in this
case?

Fixing it to honor the strict "block scoping" concept is not hard, now
that the scoping level is honored from Lua too (see

https://github.com/rpm-software-management/rpm/commit/1767bc4fd82bfacee622e698f9f0ae42c02126fa).

In this case the above reproducer would emit

%foo
true
%foo



As far as I can tell, usage of %define is discouraged and I never
really
understood why. But this example explains it and this should be the
right behavior IMO.



Yup, people are told to avoid %define like it was the plague, which
is just
plain bullsh**. You just need to understand the scoping of macros. Which
almost nobody does, no matter how many times it gets repeated here or
there.
That the rpm implementation of the scoping is very buggy doesn't exactly
help of course.

So I've really started thinking it's better to make rpm behave the way
people expect it to instead of banging my head against this for
another 10+
years.



For what it's worth, Andreas Scherer's debbuild[1] treats %define just
like %global in terms of scope, which I guess is the expectation of
most people (though that might be because it's not exactly well
documented that there *are* rules for %define scoping). That said, if
the scoping rules are clear (and work as they're supposed to!), I
don't see a reason why not to actually have scoping for %define.

[1]: https://github.com/ascherer/debbuild



%define inside a parametric macro is scoped and almost correctly
enforced too in all existing rpm versions. Ditto for the automatic
macros (%*, %1, %2 etc) inside parametric macros.

So what I'm leaning towards to is a simple two-level scoping:
1) macros %define'd inside parametric macro are local to that macro,
including the automatic argument macros (%global is obviously %global
there as well)
2) everything else is globally scoped

This would mean that the common and troublesome idiom of:

"%{!?foo: %define foo 1}"

...is no more equals this useless C-construct:
if (foo) {
int foo = 1;
}

...but instead places the foo in the nearest meaningful scope: either
the global scope or the local "function" scope, in case it occurred
inside a parametric macro.

Rpm doesn't currently limit macro visibility from other scopes at all,
I'd like to change that too so that you only ever see the global scope,
and additionally within parametric functions, the macros defined in
*that* parametric scope. And nothing else, no leftovers from other
"calls" etc.


I knew I was forgetting something:

Enforcing macro visibility also absolutely requires fixing the expansion 
of parametric macro arguments (GH issue #127). But that's also one of 
the long-standing warts in the macro engine, and I'd expect it to be 
well more than worth any breakage that it may cause.


- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Fwd: [Rpm-maint] Fixing macro scoping

[Panu Matilainen]

On 02/06/2017 04:12 PM, Neal Gompa wrote:

On Mon, Feb 6, 2017 at 4:50 AM, Panu Matilainen
<pmati...@laiskiainen.org> wrote:

On 02/06/2017 10:56 AM, Vít Ondruch wrote:




Dne 3.2.2017 v 13:14 Panu Matilainen napsal(a):



Stunned silence on rpm-maint, forwarding to rpm-ecosystem in hopes of
a larger and livelier audience...

- Panu -

 Forwarded Message 
Subject: [Rpm-maint] Fixing macro scoping
Date: Mon, 23 Jan 2017 12:30:21 +0200
From: Panu Matilainen <pmati...@laiskiainen.org>
To: rpm-ma...@lists.rpm.org <rpm-ma...@lists.rpm.org>


Consider the following snippet, originating from
https://bugzilla.redhat.com/show_bug.cgi?id=552944:

%{!?foo: %define foo bar}
%define dofoo() true
echo 1: %{foo}
%dofoo
echo 2: %{foo}

I'd assume everybody agrees both %{foo}'s should evaluate to the same
value, but that is not the case currently. Using a cli-variant of the
above:

[pmatilai@sopuli rpm]$ rpm --define 'dofoo() true' --eval '%{!?foo:
%define foo bar}' --eval '%foo' --eval '%dofoo' --eval '%foo'
warning: Macro %foo defined but not used within scope

bar
true
%foo

The flaw here is that rpm supposedly implements block level scoping
for macros (so in the above example, "foo" would only exist in the
{!?foo:...} block), but doesn't actually enforce that, unless a
parametric macro is "called". Current rpm warns about it, but warnings
or not this behavior doesn't make the slightest sense.

The question is, what do you think %{foo} should evaluate to in this
case?

Fixing it to honor the strict "block scoping" concept is not hard, now
that the scoping level is honored from Lua too (see

https://github.com/rpm-software-management/rpm/commit/1767bc4fd82bfacee622e698f9f0ae42c02126fa).
In this case the above reproducer would emit

%foo
true
%foo



As far as I can tell, usage of %define is discouraged and I never really
understood why. But this example explains it and this should be the
right behavior IMO.



Yup, people are told to avoid %define like it was the plague, which is just
plain bullsh**. You just need to understand the scoping of macros. Which
almost nobody does, no matter how many times it gets repeated here or there.
That the rpm implementation of the scoping is very buggy doesn't exactly
help of course.

So I've really started thinking it's better to make rpm behave the way
people expect it to instead of banging my head against this for another 10+
years.



For what it's worth, Andreas Scherer's debbuild[1] treats %define just
like %global in terms of scope, which I guess is the expectation of
most people (though that might be because it's not exactly well
documented that there *are* rules for %define scoping). That said, if
the scoping rules are clear (and work as they're supposed to!), I
don't see a reason why not to actually have scoping for %define.

[1]: https://github.com/ascherer/debbuild



%define inside a parametric macro is scoped and almost correctly 
enforced too in all existing rpm versions. Ditto for the automatic 
macros (%*, %1, %2 etc) inside parametric macros.


So what I'm leaning towards to is a simple two-level scoping:
1) macros %define'd inside parametric macro are local to that macro, 
including the automatic argument macros (%global is obviously %global 
there as well)

2) everything else is globally scoped

This would mean that the common and troublesome idiom of:

"%{!?foo: %define foo 1}"

...is no more equals this useless C-construct:
if (foo) {
int foo = 1;
}

...but instead places the foo in the nearest meaningful scope: either 
the global scope or the local "function" scope, in case it occurred 
inside a parametric macro.


Rpm doesn't currently limit macro visibility from other scopes at all, 
I'd like to change that too so that you only ever see the global scope, 
and additionally within parametric functions, the macros defined in 
*that* parametric scope. And nothing else, no leftovers from other 
"calls" etc.


That'd seem to me like quite a clear an simple set of rules, that 
packagers etc are likely to intuitively get more or less right because 
it actually resembles scoping in other simple scripting languages.


One open question I have is what to do with %undefine's: currently rpm 
allows %undefining anything from any scope, and that is at odds with any 
attempt to rationalize and formalize the scoping to something actually 
comprehensible. A simple approach is that you can only undefine 
something from your local scope or the global scope. But what if there's 
something by the same name in between?


- Panu -

- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Fwd: [Rpm-maint] Fixing macro scoping

[Panu Matilainen]

On 02/06/2017 10:56 AM, Vít Ondruch wrote:



Dne 3.2.2017 v 13:14 Panu Matilainen napsal(a):


Stunned silence on rpm-maint, forwarding to rpm-ecosystem in hopes of
a larger and livelier audience...

- Panu -

 Forwarded Message 
Subject: [Rpm-maint] Fixing macro scoping
Date: Mon, 23 Jan 2017 12:30:21 +0200
From: Panu Matilainen <pmati...@laiskiainen.org>
To: rpm-ma...@lists.rpm.org <rpm-ma...@lists.rpm.org>


Consider the following snippet, originating from
https://bugzilla.redhat.com/show_bug.cgi?id=552944:

%{!?foo: %define foo bar}
%define dofoo() true
echo 1: %{foo}
%dofoo
echo 2: %{foo}

I'd assume everybody agrees both %{foo}'s should evaluate to the same
value, but that is not the case currently. Using a cli-variant of the
above:

[pmatilai@sopuli rpm]$ rpm --define 'dofoo() true' --eval '%{!?foo:
%define foo bar}' --eval '%foo' --eval '%dofoo' --eval '%foo'
warning: Macro %foo defined but not used within scope

bar
true
%foo

The flaw here is that rpm supposedly implements block level scoping
for macros (so in the above example, "foo" would only exist in the
{!?foo:...} block), but doesn't actually enforce that, unless a
parametric macro is "called". Current rpm warns about it, but warnings
or not this behavior doesn't make the slightest sense.

The question is, what do you think %{foo} should evaluate to in this
case?

Fixing it to honor the strict "block scoping" concept is not hard, now
that the scoping level is honored from Lua too (see
https://github.com/rpm-software-management/rpm/commit/1767bc4fd82bfacee622e698f9f0ae42c02126fa).
In this case the above reproducer would emit

%foo
true
%foo


As far as I can tell, usage of %define is discouraged and I never really
understood why. But this example explains it and this should be the
right behavior IMO.


Yup, people are told to avoid %define like it was the plague, which is 
just plain bullsh**. You just need to understand the scoping of macros. 
Which almost nobody does, no matter how many times it gets repeated here 
or there. That the rpm implementation of the scoping is very buggy 
doesn't exactly help of course.


So I've really started thinking it's better to make rpm behave the way 
people expect it to instead of banging my head against this for another 
10+ years.


- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Fwd: [Rpm-maint] Fixing macro scoping

[Panu Matilainen]

On 02/03/2017 04:54 PM, Panu Matilainen wrote:

On 02/03/2017 03:26 PM, Pascal Terjan wrote:

On 3 February 2017 at 12:14, Panu Matilainen
<pmati...@laiskiainen.org> wrote:


Stunned silence on rpm-maint, forwarding to rpm-ecosystem in hopes of a
larger and livelier audience...

- Panu -

 Forwarded Message 
Subject: [Rpm-maint] Fixing macro scoping
Date: Mon, 23 Jan 2017 12:30:21 +0200
From: Panu Matilainen <pmati...@laiskiainen.org>
To: rpm-ma...@lists.rpm.org <rpm-ma...@lists.rpm.org>


Consider the following snippet, originating from
https://bugzilla.redhat.com/show_bug.cgi?id=552944:

%{!?foo: %define foo bar}
%define dofoo() true
echo 1: %{foo}
%dofoo
echo 2: %{foo}

I'd assume everybody agrees both %{foo}'s should evaluate to the same
value,
but that is not the case currently. Using a cli-variant of the above:

[pmatilai@sopuli rpm]$ rpm --define 'dofoo() true' --eval '%{!?foo:
%define
foo bar}' --eval '%foo' --eval '%dofoo' --eval '%foo'
warning: Macro %foo defined but not used within scope

bar
true
%foo

The flaw here is that rpm supposedly implements block level scoping for
macros (so in the above example, "foo" would only exist in the
{!?foo:...}
block), but doesn't actually enforce that, unless a parametric macro is
"called". Current rpm warns about it, but warnings or not this behavior
doesn't make the slightest sense.

The question is, what do you think %{foo} should evaluate to in this
case?

Fixing it to honor the strict "block scoping" concept is not hard,
now that
the scoping level is honored from Lua too (see
https://github.com/rpm-software-management/rpm/commit/1767bc4fd82bfacee622e698f9f0ae42c02126fa).

In this case the above reproducer would emit

%foo
true
%foo

Another option is slightly changing the whole scoping notion: parametric
macros require locally scoped macros for the automatic argument
macros like
%#, %* and %1 anyway (it's flawed too currently, see below). So
perhaps the
macro scoping should follow the current "call level", ie a macro defined
inside a parametric macro body is local to that macro, and everything
else
is global. In this case the reproducer would emit
bar
true
bar


My first reaction was to prefer this one because '%{!?foo: %define foo
bar} has been used in tens of thousands of packages and sort of worked
"forever"
But replacing define with global is not very hard so no strong opinion...


Me too, but no strong opinion either. Also I'm biased, because


Oops. What that was supposed to say is that I'm biased because for years 
I've been explaining this "implemented before my time" block scoping 
thing to baffled packagers, so changing it now would feel at least 
somewhat awkward or something. But ultimately I just want a sane, 
clearly defined behavior, never mind my feelings :)



___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

[Rpm-ecosystem] Fwd: [Rpm-maint] [RFC] Move all informative messages to stderr by default (ticket #31)

[Panu Matilainen]

Forwarding to rpm-ecosystem for a wider audience...

- Panu -

 Forwarded Message 
Subject: [Rpm-maint] [RFC] Move all informative messages to stderr by 
default (ticket #31)

Date: Thu, 13 Oct 2016 09:41:27 +0300
From: Panu Matilainen <pmati...@redhat.com>
To: rpm-ma...@lists.rpm.org

Rpm has always spewed all sorts of "informative" messages into stdout
where it gets mixed up with user requested information such as query
output, complicating scripts unnecessarily and also breaking unix
traditions.

Make the user-requested output (namely query and verify output) go to
stdout by explicitly fprintf()'ing it there, and all of rpmlog()
output goes to stderr (or other specified logfile).

Fixes rpm.org ticket #31, RhBug:1358861, RhBug:247239, RhBug:439290...
OTOH it *will* break scripts people use for querying etc. Making
scripts compatible with both old and new behavior shouldn't be hard, but
it does require changes and some *will* complain. So obviously not
4.13 material, target would be the next major release after that.

Comments, thoughts?
---
 lib/query.c| 10 +-
 lib/rpmchecksig.c  |  4 ++--
 lib/verify.c   |  4 ++--
 rpmio/rpmlog.c | 15 ---
 tests/rpmdb.at |  6 +++---
 tests/rpmverify.at |  6 +++---
 6 files changed, 15 insertions(+), 30 deletions(-)

diff --git a/lib/query.c b/lib/query.c
index 739a0de..f007d05 100644
--- a/lib/query.c
+++ b/lib/query.c
@@ -82,7 +82,7 @@ static void printFileInfo(const char * name,
(void)strftime(timefield, sizeof(timefield) - 1, fmt, tm);
 }
 -rpmlog(RPMLOG_NOTICE, "%s %4d %-8s%-8s %10s %s %s\n", perms,
+fprintf(stdout, "%s %4d %-8s%-8s %10s %s %s\n", perms,
 	(int)nlink, ownerfield, groupfield, sizefield, timefield,  	link ? 
link : name);

 free(perms);
@@ -101,7 +101,7 @@ int showQueryPackage(QVA_t qva, rpmts ts, Header h)
char *str = headerFormat(h, qva->qva_queryFormat, );
if ( str != NULL ) {
-   rpmlog(RPMLOG_NOTICE, "%s", str);
+   fprintf(stdout, "%s", str);
free(str);
} else {
rpmlog(RPMLOG_ERR, _("incorrect format: %s\n"), errstr);
@@ -199,11 +199,11 @@ int showQueryPackage(QVA_t qva, rpmts ts, Header h)
 fflags & RPMFILE_DOC ? "1" : "0",
 frdev,
 (flink && *flink ? flink : "X"));
-   rpmlog(RPMLOG_NOTICE, "%s%s\n", buf, add);
+   fprintf(stdout, "%s%s\n", buf, add);
free(add);
} else
if (!rpmIsVerbose()) {
-   rpmlog(RPMLOG_NOTICE, "%s%s\n", buf ? buf : "", fn);
+   fprintf(stdout, "%s%s\n", buf ? buf : "", fn);
}
else {
uint32_t fnlink = rpmfiFNlink(fi);
@@ -219,7 +219,7 @@ int showQueryPackage(QVA_t qva, rpmts ts, Header h)
if (now == 0)
now = time(NULL);
if (buf) {
-   rpmlog(RPMLOG_NOTICE, "%s", buf);
+   fprintf(stdout, "%s", buf);
}
printFileInfo(fn, fsize, fmode, fmtime, frdev, fnlink,
fuser, fgroup, flink, now);
diff --git a/lib/rpmchecksig.c b/lib/rpmchecksig.c
index f17a18a..41be147 100644
--- a/lib/rpmchecksig.c
+++ b/lib/rpmchecksig.c
@@ -330,10 +330,10 @@ static int rpmpkgVerifySigs(rpmKeyring keyring, 
rpmQueryFlags flags,

 res = failed;
  if (rpmIsVerbose()) {
-   rpmlog(RPMLOG_NOTICE, "%s", buf);
+   fprintf(stdout, "%s", buf);
 } else {
const char *ok = (failed ? _("NOT OK") : _("OK"));
-   rpmlog(RPMLOG_NOTICE, "%s%s%s%s%s%s%s%s\n", buf, ok,
+   fprintf(stdout, "%s%s%s%s%s%s%s%s\n", buf, ok,
   missingKeys ? _(" (MISSING KEYS:") : "",
   missingKeys ? missingKeys : "",
   missingKeys ? _(") ") : "",
diff --git a/lib/verify.c b/lib/verify.c
index 2ac7c08..f6afa806 100644
--- a/lib/verify.c
+++ b/lib/verify.c
@@ -449,7 +449,7 @@ static int verifyHeader(rpmts ts, Header h, 
rpmVerifyAttrs omitMask, int ghosts)

if (buf) {
if (fstate)
buf = rstrscat(, " (", fstate, ")", NULL);
-   rpmlog(RPMLOG_NOTICE, "%s\n", buf);
+   fprintf(stdout, "%s\n", buf);
buf = _free(buf);
}
 }
@@ -486,7 +486,7 @@ static int verifyDependencies(rpmts ts, Header h)
while ((p = rpmpsiNext(psi)) != NULL) {
char * ps = rpmProblemString(p);
-   rpmlog(RPMLOG_NOTICE, "\t%s\n", ps);
+   fprintf(stdout, "\t%s\n", ps);
free(ps);
}
   

Re: [Rpm-ecosystem] Special meaning of "+" (?) separator

[Panu Matilainen]

On 09/12/2016 05:18 PM, Panu Matilainen wrote:

On 09/12/2016 03:10 PM, Neal Gompa wrote:

On Mon, Sep 12, 2016 at 8:07 AM, Florian Festi <ffe...@redhat.com> wrote:

Changing the way + is treated in version compare is really a bad idea.
So this feature would need a new char that is currently not permitted in
versions. Candidates include: #, ^, @, §, $, ?


Other possible candidates would be |, %, &, ! and *.

Most of them clash with various shell features, making them cumbersome 
in reality, but certainly some more than others.


- Panu -

___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem

Re: [Rpm-ecosystem] Special meaning of "+" (?) separator

[Panu Matilainen]

On 09/12/2016 03:10 PM, Neal Gompa wrote:

On Mon, Sep 12, 2016 at 8:07 AM, Florian Festi  wrote:

Changing the way + is treated in version compare is really a bad idea.
So this feature would need a new char that is currently not permitted in
versions. Candidates include: #, ^, @, §, $, ?


Why is changing '+' in version comparison a bad idea? It currently
exists as a separator that does nothing, and every usage of it I've
seen even in RPM-based distros makes the assumption that it works the
way the proposed changes will actually make it function.

If anything, this would align how the operator is used in practice,
which is always a good thing, in my book.


I'm on Florians side here.

Changing longstanding existing semantics is always risky business, you 
dont know what might break in somebodys environment. Neither do I. 
There's no point taking a risk of breakage when its not needed.


- Panu -
___
Rpm-ecosystem mailing list
Rpm-ecosystem@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-ecosystem