Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

v6 will have signed payload size information : 
https://github.com/rpm-software-management/rpm/pull/3017/commits/784bb9076d614da33d29123f5ef6236a57d38463

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9324984
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

> Thing is, since the payload is typically compressed, offsets are useless 
> because they'd just point to somewhere in the middle of a compression stream, 
> you can't jump to it without reading the whole thing anyhow.

zstd frame headers can carry both compressed and uncompressed sizes, which 
makes it a little easier to seek around within the compressed payload.

> If the files in the payload were individually compressed, it'd seem quite 
> reasonable to have offsets stored. Of course that would likely loose 
> something in the compression ratio.

That's mostly what I did in my rpm-rs based poc - files aren't individually 
compressed, but the start and end of individual file data segments correspond 
to zstd frame boundaries, so individual files can be extracted or written via 
`BTRFS_IOC_ENCODED_WRITE`/reflinked without much effort.


> > The rpm payload format isn't modified, although there's a slight "bending" 
> > of the cpio/newc spec to use the filename field for padding. zstandard 
> > frames making up the compressed rpm payload explicitly carry both 
> > compressed and uncompressed lengths, to allow detection of 
> > filesystem-supported I/O sizes.
> 
> Note that the cpio/newc format is on its way out, you don't want to design 
> too much around that. In rpm v4 it's used for packages where all contained 
> files are below 4GB, but v6 will always use the newer rpm-specific format 
> which doesn't carry any file metadata in the payload: 
> https://rpm-software-management.github.io/rpm/manual/format_v6.html#payload

Returning to this, I suppose I could still pad `07070X` payloads relatively 
easily by injecting individual "padding" files into the archive. This would 
still be compatible with V4, although it might be nice to reserve a special 
*padding* path in the V6 spec which can be skipped over (where V4 would still 
extract / install it). Is the V6 draft still open for these kinds of proposals?
cc: @lnussel

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9281616
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Demi Marie Obenour]

@ddiss fsverity would also be suitable.  If you go with this approach, I 
recommend also including the total length of the payload in the (signed) 
header, to avoid vulnerabilities where extra data somehow doesn’t get hashed.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9281315
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

@DemiMarie I wonder whether packaging fsverity (Merkle tree) metadata into the 
rpm header would be an option for performant block-based hashing. It'd also 
bloat the rpm header, but may have the benefit of allowing the metadata to be 
reused for post-installation integrity and authenticity protection (assuming 
the files remain read-only and reside on ext4/btrfs).
For my specific `BTRFS_IOC_ENCODED_WRITE` use case, I didn't intend to 
implement it as a plug-in, but as part of core. The implementation doesn't 
require modifications to the existing on-disk format, and the only FS specific 
behaviour is the actual encoded write ioctl calls.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9280921
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

I think as long as the zstd frame boundaries align with individual file data 
segments then it should still work fine with `BTRFS_IOC_ENCODED_WRITE` and 
reflink, although we'd need to write out individual files immediately rather 
than staging the complete cpio archive.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9275970
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

Thing is, since the payload is typically compressed, offsets are useless 
because they'd just point to somewhere in the middle of a compression stream. 
If the files in the payload were individually compressed, it'd seem quite 
reasonable to have offsets stored. Of course that would likely loose something 
in the compression ratio.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9271027
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

There are no offsets stored, so I don't think there is.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9271000
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

Is there any way to pad (for alignment) between file data segments in the v6 
payload? IIUC, the rpm header carries file data payload offset and length 
information, so perhaps sparse / zero ranges in between might be possible?

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9270904
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

> The rpm payload format isn't modified, although there's a slight "bending" of 
> the cpio/newc spec to use the filename field for padding. zstandard frames 
> making up the compressed rpm payload explicitly carry both compressed and 
> uncompressed lengths, to allow detection of filesystem-supported I/O sizes.

Note that the cpio/newc format is on its way out, you don't want to design too 
much around that. In rpm v4 it's used for packages where all contained files 
are below 4GB, but v6 will always use the newer rpm-specific format which 
doesn't carry any file metadata in the payload: 
https://rpm-software-management.github.io/rpm/manual/format_v6.html#payload

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9270749
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

Makes sense, thanks for the clarification.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9270632
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

The compressed and uncompressed digests are *alternatives*, both cannot be 
valid at the same time. Rpm calculates both but uses the one that matches (if 
any). This is to allow freely changing between uncompressed and compressed 
format of the same content, which "obviously" must be a legit case.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9270613
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

One other thing I noticed is that the rpm header carries the digest of the 
compressed payload in addition to the uncompressed payload digest. Verification 
of the compressed payload alongside `BTRFS_IOC_ENCODED_WRITE` is relatively 
straightforward, but verifying the uncompressed payload would require 
decompression, which we're obviously trying to avoid with this optimization.
I think it's safe to only perform compressed payload verification - zstd frames 
can carry an optional xxHash64 hash, which should be suitable for correctness 
checking if desired.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9270483
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

If what you mean is that the payload shouldn't be used before it's been checked 
against a digest in the cryptographically verified header, then yes. The 
question is what quantifies as "used".
The payload data needs to go somewhere while we're calculating the digest, in 
which case we could "stream" it to Btrfs in it's compressed, yet-to-be-verified 
form via `BTRFS_IOC_ENCODED_WRITE`. However, doing so would allow any 
concurrent (pre-verification) reads to effectively fuzz the btrfs/zstd 
decompression code path.
One protection mechanism might be to write out the payload in its regular 
(compressed) form before attempting to *switch* those regular extents to 
compressed extents on btrfs via some new `copy_file_range()` flag. The 
regular->compressed extent switch would only be attempted following successful 
payload verification.
I'm very much open to any other suggestions :-)

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9270320
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Demi Marie Obenour]

Do you plan on doing streaming cryptographic verification?  See 
.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9269372
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

I hacked together a proof of concept implementation which uses 
`BTRFS_IOC_ENCODED_WRITE` to write a zstd-compressed cpio payload directly to 
disk as-is, from a carefully aligned rpm. Compressed extents are then reflinked 
to the installation path.
presentation: 
https://www.youtube.com/live/qNGw8IKaqc0?si=vrLkk8Bi9Odfqm4Z=10325
rpm-rs based source: 
https://github.com/ddiss/rpm/tree/poc-btrfs-zstd-encoded-io-reflink-extract

The rpm payload format isn't modified, although there's a slight "bending" of 
the cpio/newc spec to use the filename field for padding. zstandard frames 
making up the compressed rpm payload explicitly carry both compressed and 
uncompressed lengths, to allow detection of filesystem-supported I/O sizes.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-9269013
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

> To add to this: Plugins should not get access to content that has not been 
> verified yet. 

Regardless of the technical requirements, this is a fine example of what a sane 
API looks like: it's not about just making X somehow possible, but building the 
right usage into the API. Often easier said than done of course.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-5061162
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Richard Phibel]

Thanks a lot for these comments. I think I have a better idea of what I need to 
do. I will work on a new implementation.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-5055827
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Demi Marie Obenour]

To add to this: Plugins should not get access to content that has not been 
verified yet.  That means creating a new method of cryptographic verification, 
one that allows streaming verification of the data.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-5054477
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Panu Matilainen]

Okay so, trying to give a better idea of what I'm after:

The fact that RPMRC_PLUGIN_CONTENTS is still needed is a strong signal that 
it's not quite right. 
The way I'm imagining this is that you have a probe function for the content 
handler, that gets called for each and every package, and for stock rpm 
content, it just returns an internal function. For other content, it returns 
the handler from the first plugin that claims the content type and if nothing 
handles it, it's obviously an error. This should happen at the stage where the 
transaction is populated, not during it. 

The fsm code for handling different types doesn't differ at all, it's just a 
function pointer that gets called. Ie there's no 
rpmpluginsCallFsmFileArchiveReader() or rpmpluginsCallFsmFilePrepare(), these 
are determined by the early probe. And so there are no 
rpmteSetCustomFileInstaller() or rpmteSetCustomArchiveReader() or any 
corresponding custom-call-functions either, it's always just a function pointer 
to whatever is handling the content.



-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-5046976
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Richard Phibel]

@pmatilai Hello, I was wondering if you had a chance to have a look at my 
proposal? Please let me know if this is unclear or if you need more details.

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-4762243
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[Richard Phibel]

Hello @pmatilai,

I work at Meta with @chantra and @malmond77 . I am implementing the new API for 
the RPM CoW plugin based on your comment. This is what I have implemented so 
far:

I defined 2 new fields of type `rpmPlugin` (with associated getters and 
setters) in `rpmte` structure:
- 
[`customArchiveReader`](https://github.com/rphibel/rpm/blob/8de078508954e005ba0c974389979070cfe9431d/lib/rpmte.c#L85)
- 
[`customFileInstaller`](https://github.com/rphibel/rpm/blob/8de078508954e005ba0c974389979070cfe9431d/lib/rpmte.c#L86)

When a plugin wants to register as an archive reader for a package, it sets the 
field `customArchiveReader` in the [`psm_pre` 
](https://github.com/rphibel/rpm/blob/8de078508954e005ba0c974389979070cfe9431d/plugins/reflink.c#L165)stage.
Similarly, if it wants to register as a file installer it sets the 
`customFileInstaller` field.

Then in 
[`fsm.c`](https://github.com/rphibel/rpm/blob/8de078508954e005ba0c974389979070cfe9431d/lib/fsm.c#L849),
 if these fields are set, archive reading, file installation is deferred to the 
plugin.

Here is the code:
https://github.com/rphibel/rpm/compare/cow_denylist_rebased...cow_denylist_refactoring

Could you please let me know if this is what you had in mind?

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-4633129
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

> The idea of aligning cpio metadata is very interesting. I can see how it'd 
> help initramfs building speed tremendously.
> 
> As I understand it, RPM is pretty different: the main difference is that 
> we're trying (fairly hard) not to change the normal format of rpm as found on 
> mirrors for now. There are some very interesting ideas on how to change the 
> upstream format, but in doing so, we'd render all existing servers unable to 
> read the format.

To clarify, aligning cpio data segments for *newly built* rpms shouldn't 
necessarily require any change in format. They'd continue to function the same 
as earlier cpio payload rpms, be it with some extra zero-padding.

> If we could tolerate the breakage: I'd love to experiment with 
> `BTRFS_IOC_ENCODED_WRITE` which would reduce writes down and eliminate 
> explicit decompression. For clients or filesystems without CoW support: RPM 
> could decompress and write the normal file. I was hoping encoded writes would 
> eliminate the complex path with curl -> librepo -> rpm2extents. I'm not sure 
> you could get data from the network and write encoded data to disk in one 
> pass like we're doing now. Do you have any ideas on how to resolve that 
> challenge?

I'm not too familiar with the rpm on-disk format, but I'd hoped that 
`BTRFS_IOC_ENCODED_WRITE` could be used without a change to the format, by 
having the rpm header parsed during download to determine whether the 
compressed payload could be written as-is. With a cpio payload it'd then be a 
matter of copy_file_range()ing the (optimally aligned) compressed file data 
segments into the destination during installation.

`BTRFS_IOC_ENCODED_WRITE` appears very restrictive at this stage though:
- it requires `CAP_SYS_ADMIN`, so probably isn't a viable option for 
containers, etc.
- ioctl calls need to specify both unencoded and encoded offset+length, meaning 
that we'd still need to parse rpm payload compression metadata
- the ioctl unencoded length can't exceed 128 KiB
- for zstd encoded I/Os, the ioctl data must represent "as a single zstd frame 
with the windowLog compression parameter set to no more than 17"
  - On openSUSE Tumbleweed I see some rpms currently using zstd compression 
level 19. IIUC, Fedora uses the same zstd level

> Adding cpio metadata, along with a "null" compression type could help 
> eliminate the change in `fsm.c` on how the payload is iterated. Note that 
> `rpm2extents` does not (and cannot) touch headers without invalidating 
> signatures, so the change in compression type is inferred and handled in the 
> plugin.
> 
> Lastly, there's another optimization that would be lost in adopting cpio 
> formatting: content de-duplication. I'm not sure how important this is tho in 
> the big picture, so it might be a worthwhile tradeoff.

Indeed. FWIW, I think your extent based approach offers a lot of worthwhile 
benefits, but just wanted to point out that something similarly CoW friendly 
(although less efficient) is possible without necessarily requiring invasive 
changes :-)

> Thanks for the feedback! Matthew.

Thanks for the response!


-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-4337009
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] API improvement to accommodate for RPM CoW (PR#1470) (Discussion #2057)

[David Disseldorp]

Thanks for working on this - the new CoW extent based approach looks exciting. 
I have a few comments / questions on the initial proposal...

> Files are converted (“transcoded”) locally during download using 
> /usr/bin/rpm2extents (part of rpm codebase). The format is not intended to be 
> “portable” - i.e. copying the files from the cache is not supported.
>
> Regular RPMs use a compressed .cpio based payload. In contrast, extent based 
> RPMs contain uncompressed data aligned to the fundamental page size of the 
> architecture, e.g. 4KiB on x86_64. This alignment is required for 
> FICLONERANGE to work. Only files are represented in the payload, other 
> directory entries like symlinks, device nodes etc are constructed entirely 
> from rpm header information. Files are referenced by their digest, so 
> identical files are de-duplicated.

I just wanted to highlight that, although not optimal, cpio archives can still 
be used alongside reflinks if the `newc` spec is *bent* a little to provide 
file data-segment alignment. The kernel initramfs archive format is quite 
static, so for [dracut-cpio](https://github.com/dracutdevs/dracut/pull/1531) 
(`dracut --enhanced-cpio`) reflinks I added functionality to zero-pad file 
names so that subsequent file data segments are block aligned. kernel and GNU 
cpio extractors handle this fine, with the caveat that padding can't exceed 
`PATH_MAX`.

On a separate note, Btrfs recently gained support for writing compressed 
extents directly to disk via the new `BTRFS_IOC_ENCODED_WRITE` ioctl. IIUC, 
Fedora also recently changed to using zstd by default for Btrfs root and rpms. 
Have you considered using `BTRFS_IOC_ENCODED_WRITE` functionality instead of 
performing decompression during download?

-- 
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/discussions/2057#discussioncomment-4327609
You are receiving this because you are subscribed to this thread.

Message ID: 
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint