Re: [Rpm-maint] relicensing for Maximum RPM

[Jeff Johnson]

 of the ones who worked for rht at the time they were writing it. So I
 suspect Paul and jbj's contributions are (c) Red Hat inc. But  
Matthias and

 Ville are (c) themselves, unless otherwise disclaimed.

You may suspect all you wish, but you better check dates
before making false claims in public.

Please remove anyall of my contributions to Maximum RPM.
You will never get my permission for anything based on suspicions
and idle gossip on mailing lists that I do not choose to
subscribe to.

Note that I want my privacy, please refrain from mentioning
my name on this list.


73 de Jeff
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] Keep my name out of your discussions

[Jeff Johnson]

Re:

 BTW, Jeff J. must have changed his mind at some point, because rpm5
 allows short-circuiting of -bb.

You have no right to discuss the state of my mind. In fact,
you haven't a clue.

Keep my name out of your gossipy garbage.

73 de Jeff
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

[Jeff Johnson]

My bad for not noticing.

Good that padding has been added by adding (apparently) 0x1000 to the data size:
`0060: 8ead e801    0006  1074  ...t`

Adding a "free space" tag (as @rpm5.org) provides an indication that the 
signature header is padded.

Meanwhile, the metadata header is not aligned on (say) a 0x1000 offset from 
beginning of file
`
1120:          
1130:    003e  0007  ffa0  ...>
1140:  0010   8ead e801    
1150:  0039  437e  003f  0007  ...9..C~...?
1160:  436e  0010  0064  0008  ..Cn...d
`

Conventionally using a known offset would make mmap and HTTP byte ranges easier 
to use, and
its a bit easier to remember 4096 (or 0x1000) than it is to remember 4424 (ox 
0x1148).

But having some constant offset is better than rewriting rpm package files.

(aside)
Adding padding between the metadata header and the payload might also be 
attempted to align the payload on a page boundary might be useful too.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/183#issuecomment-289805262___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

[Jeff Johnson]

Ah, you have added RPMSIGTAG_RESERVEDSPACE already which I was not expecting.

The (minor but annoying problems) with the assignment to 1008 is that 
RPMSIGTAG_RESERVEDSPACE collides with RPMTAG_INSTALLTIME, and will not sort 
correctly to the end of an immutable region data section as a "free space" tag 
through headerReload-like operations if any other RPMSIGTAG_FOO is defined. 
Meanwhile -- because of the tagno assignment collisions between signature and 
metadata headers -- its rather unlikely that any other RPMSIGTAG_FOO 
assignments will ever be needed.

The functionally equivalent RPMTAG_SIGPADDING was assigned 0x3fff for 
uniqueness and to permit assigned metadata tags (e.g. RPMTAG_PUBKEYS) to be 
more easily moved between signature and metadata headers without explicit 
remapping.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/183#issuecomment-289818402___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

[Jeff Johnson]

The recently added RPMTAG_PAYLOADDIGEST computes a digest on the compressed 
payload.

Perhaps now is the time to change the plaintext used to be the uncompressed 
rather than the compressed archive payload?

The benefit of the digest on the uncompressed archive is that the digest 
becomes immune to the compression type and format, and it might become possible 
to change the compression on binary
packages without rebuilding.

Another benefit is that PAYLOADDIGEST can be verified after installation by 
reconstructing the (usually cpio archive) to verify the payload digest. While 
PAYLOADDIGEST can also be verified by reconstructing the compressed payload 
archive, there are always timestamps and implementation changes in external 
libraries that can cause false failures. A digest on the uncompressed archive 
would be immune to compression implementation changes.

Now is the time to consider before PAYLOADDIGEST has "legacy compatibility" 
issues.

(aside)
It also would not be too difficult to change rpmio to configurably compute the 
digest on either (or even both) uncompressed or compressed plaintext by 
duplicating the digest computation on an FD_t to handle 2 bits of flags on the 
pre/post I/O buffers.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/184___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

[Jeff Johnson]

The signature header contains (optional) information to verify package 
integrity.

So new tags are added when packages are signed, and a new signature header is 
created and inserted into a *.rpm package file.

However, the insertion forces a rewrite of the metadata header and payload, 
which for a distribution signing event, can rewrite a terabyte of information.

Another flaw with a variable sized signature header is that the metadata header 
and payload are not located at a constant offset, preventing, say, HTTP byte 
ranges from directly reading the header metadata blob.

One simple solution is to add a "free space" tag to the signature header to 
reserve space for a signature (or other data) to be added later. This would 
permit the new signature header to be rewritten in-place with an adjusted "free 
space" tag instead of rewriting entire packages when signing.

I will have a patch to add a "free space" tag to a signature header for rpm.org 
soon.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/183___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Adding/moving RPMTAG_PUBKEYS to the signature header (#182)

[Jeff Johnson]

RPM package files can include public keys in the metadata header.

So in principle, a package can include the public key used to sign a package 
and verify the package signature when reading package headers. The 
RPMTAG_PUBKEYS array is also within the signed immutable region and cannot be 
altered in signed package metadata.

However, there is a chicken <-> egg problem extracting RPMTAG_PUBKEYS from the 
not-yet-verified metadata header blob in order to verify the header blob before 
loading.

Adding/moving RPMTAG_PUBKEYS to the signature header would permit a stronger 
(than RPMSIGTAG_SHA1) check on the header-only (or header+payload if you 
insist) metadata blob before loading the metadata header.

(aside)
Verifying the metadata header signature before loading also avoids the need for 
explicitly implementing stronger digests like RPMSIGTAG_SHA256: the signature 
itself determines the metadata digest, and -- unlike an additional digest 
included in the signature header which can be replaced -- is a 
cryptographically stronger verification.

(aside)
This is mostly a notification of the reason why RPMTAG_PUBKEYS is likely to 
appear soon in signature headers produced by RPM5 in order to support MANDATORY 
signature verification.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/182___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Tags in the signature header are not sorted (#181)

[Jeff Johnson]

This is the hex dump of the createrepo_c Archer-3.4.5-6.x86_64.rpm signature 
header.

RPMSIGTAG_SHA1 follows RPMSIGTAG_MD5 at offset 0xa0, forcing a sort while 
loading the signature header.
_
0060: 8ead e801    0005  0054  ...T
0070:  003e  0007  0044  0010  ...>...D
0080:  03e8  0004    0001  
0090:  03ec  0007  0004  0010  
00a0:  010d  0006  0014  0001  
00b0:  03ef  0004  0040  0001  ...@
_

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/181___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Do you have measurements of the effect of fsync()?

The last time sync/fdatasync were suggested for RPM that I am aware of was 
~2011 in Mandriva. The issue comes up every few years this century ...

Here was the effect of adding fsync/fdatasync to RPM, measure on otherwise 
identical system with a package with lots of files:

`
/* Measurements from installing kernel-source package:
 * +fsync
 *  total:   1  0.00 MB640.854524 secs
 * +fdatasync
 *  total:   1  0.00 MB419.983200 secs
 * w/o fsync/fdsatasync:
 *  total:   1  0.00 MB 12.492918 secs
 */
xx = fsync(Fileno(fsm->wfd));
`

Yes this was rotating DSADI, not SSD.

BTW, the right place to patch isn't Fclose(), which is one of the more subtle 
(or hacked, YMMV) pieces of code in RPM. Instead, do the sync before closing 
the FD_t in lib/fsm.c (or whatever replaces).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291632032___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Good if doing in fsm.h ( I haven't looked).

Meanwhile a 50x slower installation is intolerable for a package manager even 
if opt-in.

Nor can an application like RPM be expected to detect whether the underlying 
filesystem is on SSD or USB or rotational disk and adapt accordingly: there 
simply aren't the necessary API/s.

I believe that your databases are stalling, package management is a huge I/O 
load, unlike any other typical application.

Meanwhile -- depending on what statistics you report (if any) -- the issue of 
1) "hardening" install's in spite of exceptional outages 2) preventing "write 
starvation" of other running applications by introducing a resource cap on RPM 
I/O, will be clear.

What measurements are you using to justify your fsync patch? That is not yet 
clear ...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291698648___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

FYI: several notes (after looking at the patch)

1) The measurements I cited above are using mmap(2) to decompress directly into 
pages (avoiding a buffer copy) marked with MADV_DONTNEED (which permits 
delayed/larger writes) to avoid a buffer copy. In addition, there is a check on 
file size of <= 128Mb to avoid consuming too many pages on extremely large 
files.

However, I have no reason to believe that mmap(2) instead of a loop over 
buffered writes will change the basic observation(s) that package management is 
a huge I/O load, and that fsync(2) will schedule large amounts of I/O that can 
starve/stall other running processes..

I suggest testing on a moderately large package with 3000+ files (like kernel  
source) to see the affect on package management. Slowing down package 
management (by perhaps a factor of 50x) will most certainly use fewer 
resources, and not compete with other running processes, as a side effect of 
calling fsync(2), but is perhaps not the best implementation. YMMV.

2) Instead of adding Fsync to walk the FD_t stack, fsync'ing every fdno, there 
is already Fileno() to return the topmost fdno of a FD_t, which is almost 
certainly all that is needed.

Using fsync(Fileno(fd)) (or equivalent: Fileno() has negative error returns) 
either at the end of the loop in lib/rpmfi.c rpmfiArchiveReadToFilePsm(), or 
(equivalently) in the parent caller, achieves the same fsync without pretending 
that Fsync() "works" with all possible fdno's.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291768318___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

For completeness (and humor ;-), there is also O_DIRECT. See 
[https://lkml.org/lkml/2007/1/10/233](url)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291777802___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

There is a modestly obscure means to attach *_DONTNEED (to achieve kernel 
delayed writes and removing pages when I/O is complete) using the existing 
rpm.org rpmfiArchiveReadToFilePsm() Fwrite->write(3) loop calling setvbuf(3) 
immediately after Fopen on the fopencookie(3) FILE *.

(aside)
Personally, I think mmap(2) and madvise(2) are easier to implement/maintain 
than rpmio fopencookie(3) wrapped stdio(3) abstractions like setvbuf(3): YMMV.

The idea behind *_DONTNEED -- to minimize competition for scarce resources -- 
is to hint to the kernel that RPM write buffers (actually pages) can be reused 
by other processes as soon as written to disk, rather than cached. RPM package 
installation is always fire-and-forget for package files, not true in general 
for file writes.

Letting the kernel manage its own resources as necessary may be sufficient to 
prevent database stalling/starvation as an alternative to attempting explicit 
fsync(3) invalidation of cached I/O buffers
which has the unfortunate side effect of blocking RPM until the fsync(3) system 
call returns.

(aside)
This *IS* what used to be in RPM, and was a significant win at the time (but 
also included directly uncompressing into the mmap(2) buffer avoiding a copy, 
and increasing the zlib buffer size from 32K -> 256K, which has long since been 
upstreamed). Someone else will have to speak to why mmap(2) was ripped out at 
rpm.org.

(hth, off to bed)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291795090___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

There is an alternative (but more complicated) lazy implementation that might 
be attempted by doing a fsync in Fwrite (or fdWrite) when the accumulated I/O 
passes a certain mark. H ...

A general solution to add a resource cap on all rpmio I/O could also be 
attempted (but isn't easy). Berkeley DB does something like this with mpool 
file handles, and it would not be impossibly hard to use aio_write(3) instead 
of write(2) as a new type of rpmio. But that would require a common buffer 
allocation pool (or buffer-in-flight mapping) underneath rpmio (to provide the 
resource cap), as well as a reliable means to determine when I/O has actually 
been written to DASD (to free/unmap the buffer-in-flight).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291774478___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Hmmm ... glibc  (which uses mmap(2) for stdio if requested) would benefit from 
being able to mark the stdio buffer-in-use with *_DONTNEED using 
madvise/posix_fadvise.

At a minimum, cvtfmode() in rpmio/rpmio.c would need to be extended to pass the 
glibc-peculier 'm' flag through to fopen(3)  ah "info libc" indicates 
mmap(2) is permitted only for input files, never mind.

(aside)
Note that rpmio/rpmio.c cvtfmode() likely should be modified to pass all the 
glibc-peculier flags through to fopen(3): 'c', 'e', 'm', 'x' (see "info libc" 
and search for "open"). todo++ @rpm5.org

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291782667___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Another issue is that certain file systems (NFS? DFS? I forget ...) also delay 
I/O operations until close(2).

What I recall is that an application MUST check the return code of close(2) in 
order to ensure that the I/O was actually performed: it isn't sufficient to 
check the return code of write(2).

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-291775574___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

[Jeff Johnson]

(answering my own question)

Short answer: digesting compressed payload instead of compressing the digested 
payload is expedient.

Long answer: RPM goes to some lengths to preserve 
streaming-without-peeking-at-magic for *.rpm packages.

The usual RPM approach is/was to add/save tags in the metadata header to 
determine choices like format/signature/digest/compression algorithm (and 
parameters) into the metadata header. So doing a digest on the compressed 
payload permits digest/compression to be more easily setup before examining 
magic in the plaintext blob for RPM.

There are  consequences for other applications that wish to manipulate the rpm 
file format that do not use the upper layers of the RPM api where the choices 
can be buried. The consequences  are added application complexity, with a 
continual application churn to handle Newer! Better! Bestest! features like 
RPMTAG_PAYLOAD verification, and a need to pretend "legacy compatibility" with 
disablers while waiting for application implementations.

The ripple effect of a change like RPMTAG_PAYLOADDIGEST (or rpmarchive/tar 
payloads, or ECC signatures) throughout the rpm tool chain takes years, and one 
(at least me ;-) wonders whether the work is even useful if it takes years to 
achieve widely deployed implementations.

(aside)
I think a better approach going forward in RPM will be to use 
certificates/signatures (which buries the choices of signature/digest algorithm 
in "standard" ways), and to add API's to select one-of-N choices (like 
archive/compression format) lazily using magic in the objects, rather than 
adding explicit metadata tags.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/184#issuecomment-291007573___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Colorized rpmlog() output (+PATCH) (#198)

[Jeff Johnson]

SSIA

Have fun!

[rpm+color.patch.gz](https://github.com/rpm-software-management/rpm/files/915427/rpm.color.patch.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/198___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: option for initial-only content in files (#152)

[Jeff Johnson]

Add

* update by patching

and make the policies platform and machine specific.

Then call the implementation something catchier than ansible or DevOps.

Seriously: configuration != package management.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/152#issuecomment-293591026___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

@jayzmh: major invasive change to ... RPM? or something else?

I'll happily write the code for rpm.org RPM if you wish. I fail to see how 
reverting to what RPM used to do (i.e. use mmap with MADV_DONTNEED and most 
importantly for you using msync(MS_ASYNC) which schedules flush to disk like 
fsync freeing resources but returns immediately permitting RPM to continue its 
software updating.

There's another approach using O_SYNC/O_DSYNC which I've not looked at because 
(correctly) added to POSIX and linux in 2008, which is long after I 
looked/measured the original mmap implementation. But lets not go to 
O_SYNC/O_DSYNC until MADV_DONTNEED is measured please.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292327785___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

@jayzmh - add a static variable to expand the macro as a "one shot" ... there 
are many places in rpm that do that, hurts little (until rpm can capture global 
configuration object).

Meanwhile I recommend the *_DONTNEED mmap implementation instead of fsync. RPM 
used that for years, and *_DONTNEED was originally suggested by Ben LaHaise (a 
linux kernel developer) as the right implementation (particularly for what you 
want): let the kernel manage its resources, don't try to devise some 
configurable opt-in policy permitting an application using fsync to second 
guess (or permit sysadmins the pretense of second guessing) kernel resource 
management algorithms.

Meanwhile: do you have measurements that you can report?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292091677___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Make 'rpm -V' more resistent against rpmdb manipulations (#196)

[Jeff Johnson]

Hmmm ... its not clear what  exploit is used (from just reading the file at the 
URL you gave).

The RPM versions (4.1/4.2) mentioned in the URL are quite ancient (circa RHL 
8/9).

Even on those ancient systems, /var/lib/rpm requires access to root:root (or 
rpm:rpm) in order to modify an rpmdb (but the derivative task of the exploit is 
hiding a rootkit, where root:root has already been achieved, by changing 
information in the rpmdb).

The provision in RPM for careful rootkit forensics is to use "rpm -Vp ..." from 
a CDROM (or other offline/immutable media).

Hardening the manifest of installed packages (i.e. what is in an rpmdb) on an 
already exploited system would require a security protocol to 
authenticate/authorize rpm installations, most likely using a TPM and a trusted 
boot to ensure that the RPM executable was not compromised, and then to keep 
track of the list of package headers which SHOULD be installed, as well as 
verifying header signatures before verifying package contents.

This isn't an easy problem to solve.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/196#issuecomment-292833133___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] multiple signatures support? (#189)

[Jeff Johnson]

There is nothing stopping other applications from using detached signatures on 
*.rpm files as necessary. RPM cannot carry one (or multiple) signatures within 
signed plaintext.

(aside)
In principle a different ping-pong like signing could be attempted to ensure 
that both signature/metadata headers are signed with different pairs of keys, 
with the pubkey(s) that signed the signature header in the metadata header and 
vice versa, but lets not go there please)

The core issue here seems to be hardlinking *.rpm files between different 
distributions, where the packages are identical except for the signature using 
different keys, and therefor hard linking is impossible.

Having multiple signatures only solves one part of the puzzle: making the *.rpm 
content static so that files can be hard linked by including multiple 
signatures.

The signing as well as the verification becomes far more complex because of the 
key management involved associating multiple keys and signatures where needed, 
particularly if RPM needs a policy file to specify which signature needs to be 
verified.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/189#issuecomment-292839459___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

OK, here is one way to install a file without leaving the file in the cache.

The simplest explanation is to show the --rpmiodebug log (interspersed with 
some debugging):

*** Fopen ufdio path /usr/share/doc/hello-2.0/README;58ea9807 fmode w+.ufdio
*** ufdOpen(/usr/share/doc/hello-2.0/README;58ea9807,0x242,0666)
==> Fileno(0x1c2c260) rc 31 | ufdio 31 fp (nil)
*** Fdopen(0x1c2c260,w+.ufdio)  | ufdio 31 fp (nil)
==> Fdopen(0x1c2c260,"w+.ufdio") returns fd 0x1c2c260   | ufdio 31 fp (nil)
==> Fopen("/usr/share/doc/hello-2.0/README;58ea9807",242,0666)  | ufdio 
31 fp (nil)
==> Ferror(0x1c2c260) rc 0  | ufdio 31 fp (nil)
==> Fileno(0x1c2c260) rc 31 | ufdio 31 fp (nil)
==> Fread(0x1c2fff0,0x7ffd4dedc7d0,39) rc 39| gzdio 0x1c29f20 fp 30 
| fdio -1 fp (nil)
==> Fwrite(0x1c2c260,0x7ffd4dedc7d0,39) rc 39   | ufdio 31 fp (nil)
==> Ferror(0x1c2c260) rc 0  | ufdio 31 fp (nil)
*** fdatasync(31)
*** fbefore: 1
*** posix_fadvise(31, 0, 0, 0x4)
*** fsync(31)
*** fafter: 0
ufdio:   1 writes,   39 total bytes in 0.13 secs
`

Basically there are 3 pieces to an RPM install implementation that does not 
blow out the system cache.

1) After Fwrite(), a Flush() followed by fdatasync(2) is needed.

After writing all the bytes, one page is cached (see mincore(2), the "fbefore: 
1" line).

2) posix_fadvise(POSIX_MADV_DONTNEED) is called to hint that the file need not 
remain in cache.
3) fsync is called to write the file metadata to disk.

After syncing all the bytes, no pages are cached (the "fafter: 0" line).

This is basically what rsync-fadvise is doing here: 
[https://github.com/MageSlayer/rsync-fadvise](url)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292812018___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Here is an updated patch, including the bare minimum changes with a 
%_periodic_fsync "opt-out" macro to the "production" 
rpmfiArchiveReadToFilePsm() to use fdatasync+posix_fadvise+fsync.

I've left all the debugging gook in the patch to document the --rpmiodebug 
trace and to illustrate the mincore(2) etc verification testing in the previous 
comment.

The important & necessary parts of the patch should be obvious.

The patch passes "make check" and isn't going to kill your kittens or steal 
your girl friend. ;-)

Experience with the cost of always (including rotating DASD) installing with 
fsync is needed before undertaking further steps.

[rpm+fsync.patch.gz](https://github.com/rpm-software-management/rpm/files/908578/rpm.fsync.patch.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292816086___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

As promised, here is a proof-of-concept patch to permit using either 
fwrite+fsync or mmap+madvise+msync while writing files onto the file system 
from a *.rpm package.

The patch passes "make check" (with the current known failures) on tip.

There are various switches to enable/disable the added functionality whose 
usage should be fairly obvious. I have not bothered with either build/runtime 
configuration with macros etc because its not yet clear whether msync(MS_ASYNC) 
actually solves any problem yet.

If the proof-of-concept patch using msync(MS_ASYNC) does work, then we can 
proceed from there.

[rpm+mmap.patch.gz](https://github.com/rpm-software-management/rpm/files/907882/rpm.mmap.patch.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292753264___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

I knew if I wrote a patch, better answers would instantly appear.

There is fadvise(FADV_DONTNEED) wrto fsync(2) analogous to 
madvise(MADV_DONTNEED) wrto msync(2). I will add to my patch in the next few 
days for measurement, todo++.

The easy answer for your "Chef installs packages and my databases stall!" is 
likely to run
use nocache from [https://github.com/Feh/nocache](url). There's also a means to 
use a cgroup there to put a resource cap on RPM (or any other application) 
memory usage.

Perhaps the group scripting can be use with rpm lib: todo++.

Meanwhile rsync backups have a similar effect to your RPM issue. Both rpm/rsync 
have huge I/O needs that easily cause cache blowout. Recent rsync has not only 
a solution, but rsync can also use mincore(2) to explicitly verify that no 
blocks/pages from a file remain in the cache. Ditto, todo++ to add to patch ...


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292762306___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding high precision event time stamps to RPM (#197)

[Jeff Johnson]

Attached is patch to extend RPMTAG_{BUILDTIME,INSTALLTIME,INSTALLTID} to add a 
2nd element that contains a struct timespec tspec.tv_nsec 2nd element.

There are 2 new getters/setters added to the rpmts API: rpm{Get,Set}Tid2().

About the only subtlety (so far) is the need (for rpmdb "legacy compatibility") 
to index only tspec.tv_secs in RPMDBI_INSTALLTID. A better index could be 
attempted, but the value of the Installed index w/o --rollback is questionable. 
Using a UUIDv1 timestamp would be one possible choice for a more precise 
RPMDBI_INSTALLTID key.

These are the reasons (that I know of) to increase the precision of time stamps:

0) time(2) is very last century (but still doing the job).

1) INSTALLTIME can be inaccurate these days: fast processors can install 
multiple packages/second.

2) INSTALLTID was designed to be unique: adding microsecs/nanosecs helps ensure 
the uniqueness.

3) UUIDv1 time stamps are likelier to be unique if generated with a high 
precision time stamp.

[rpm+tstamp.patch.gz](https://github.com/rpm-software-management/rpm/files/912203/rpm.tstamp.patch.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/197#issuecomment-293156091___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Yes, with about 20 other similar articles and implementations, some even 
accurate.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-294249350___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Wall clock time measurements for an RPM command (this is RPM5, but the I/O 
trace is identical to rpm.org):

`sudo /usr/bin/time /opt/local/bin/rpm -Uvh --nodeps --noscripts --relocate 
/=/opt/local/tmp /var/tmp/kernel-core-4.11.0-0.rc6.git2.1.fc27.x86_64.rpm`

BASE

2.26user 0.43system 0:03.90elapsed 69%CPU (0avgtext+0avgdata 196708maxresident)k
0inputs+75744outputs (1major+82590minor)pagefaults 0swaps

2.36user 0.44system 0:07.54elapsed 37%CPU (0avgtext+0avgdata 195668maxresident)k
45280inputs+75784outputs (10major+82618minor)pagefaults 0swaps

USING fallocate+fdatasync+fadvise+fsync
---
2.60user 0.89system 2:01.76elapsed 2%CPU (0avgtext+0avgdata 196632maxresident)k
0inputs+76488outputs (1major+82685minor)pagefaults 0swaps

2.59user 0.85system 2:01.93elapsed 2%CPU (0avgtext+0avgdata 196456maxresident)k
0inputs+75752outputs (1major+82684minor)pagefaults 0swaps

2.80user 0.89system 2:30.68elapsed 2%CPU (0avgtext+0avgdata 196004maxresident)k
44952inputs+76520outputs (19major+82666minor)pagefaults 0swaps

2.77user 0.95system 3:00.88elapsed 2%CPU (0avgtext+0avgdata 196216maxresident)k
50008inputs+75792outputs (27major+82668minor)pagefuls 0swaps

(aside)
These measurements were taken on a build server -- the slower measurements were 
the result of active builds in docker.

So fallocate+fdatasync+fadvise+fsync is (nominally) ~30x slower (using a wall 
clock measured install on rotating media).

@jayzmh: (to summarize, I have said most of this before indirectly. JMHO, YMMV)

I'd like to hear simple measurements of the effect of RPM+patch on both SSD and 
rotating media. The measurements (or at least a warning) should be included 
with any "opt-in" macro that you choose for configuration so that other users 
might be forewarned of the significant delay to expect from using fsync(2).

(aside)
I'll be happy to run these measurements for you on whatever patch you choose to 
merge into RPM, as well as the tests below.

I'd also like to hear that RPM+whatever_patch_you_use "works" because the 
installed files are not in cache, not because RPM is running ~30x (or whatever 
delay on SSD) slower and hence not competing for cache effectively. There's 
nothing in the fsync(2) man page that specifies the effect on kernel cache, 
only that fsync(2) returns when file info has been written to disk.

Basically I'd like to see what mincore(2) has to say about 
installed-file-in-cache after RPM exits if you only wish to use fsync(2), or 
(perhaps) a comparison when fsync(2) is replaced with a ~50ms delay per file 
(what I am seeing as the approx fdatasync(2) delay for every file installed) if 
only
`usleep(5);`
is used.

One simple way to check the kernel cache would be to use fincore(1) from linux 
ftools here [https://code.google.com/p/linux-ftools/](url) immediately after an 
RPM install (or use the C routine fmincore() in my patch above).

@cgwalters: Jens Axboe's "buffer bloat" patch deals only with autoscaling to 
prevent large writes from degrading overall system performance. The other part 
of the problem is that file durability using fsync(2) is costly, and there 
isn't an obvious solution (at least that I can find) to how to mitigate that 
cost. Perhaps spinning off a helper thread on file close will "work": I'm not 
sure I trust the widely cited conclusions here 
[http://oldblog.antirez.com/post/fsync-different-thread-useless.html](url), 
particularly with the "buffer bloat" patch. RPM I/O is also many files, not one 
very large file.

@pmatilai: worrying about naming a macro "_periodic_fsync" and later 
generalizing to call f*sync(2) to be, say, every nBytes/nSecs is likely 
unnecessary. For starters, RPM installations are many files, not a single large 
file, and fsync(2) takes a single fdno argument. Doing "w+.fdio" (i.e. open RW 
and truncating), and pre-sizing using fallocate(2) (so that st->st_size 
metadata does not need to be repeatedly sync'd as writes progress) is likely a 
trivial implementation in the right direction on all media.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-294409265___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

@pmatila: if you *do* attempt to configure +/- fsync(2) somehow, then 
durability is likely the better approach. By "durability", I mean "this file is 
essential to booting" (and hence is worth paying the cost of fsync(2) to ensure 
that the file is correctly written to disk).

Some simple patterns could be applied to paths to choose fsync(2) (or not). One 
could also add a %sync flag in spec files to set a flag for per-file-durability 
under packager control.

OTOH, per-file != per-package or per-transaction durability. But at least 
per-package atomicity can be achieved by delaying the FILE.TID -> FILE rename 
into place. Unclear whether its worth the effort *shrug*.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-294410365___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Yes, fdatasync+fadvise_syncfs removes pages from system cache:

`
--> Fincore(0x61408e40) fdno 16 path 
/opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f7761a
<-- Fincore(0x61408e40) fdno 16 rc 0
<-- fdClose(0x61408e40) rc 0| UFD -1 fp 0x61204e40
 FDIO:  30write(s),  3763601 total bytes in 0.003806 secs
 FDIO:  32   digest(s),  3763601 total bytes in 0.037065 secs
 FDIO:   1alloc(s),  3763601 total bytes in 0.40 secs
 FDIO:   1 datasync(s),0 total bytes in 0.072015 secs
 FDIO:   1   syncfs(s),0 total bytes in 0.063909 secs
 FDIO:   2   incore(s),  3764224 total bytes in 0.000263 secs
`

I'm not sure whether per-partition rather than per-file durability is usefully 
needed for RPM.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-295296195___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

A measurement of the impact is needed first, enabler/disabler configuration can 
always be added.

In general, RPM does best with "opt-out", not "opt-in" policies. There may be 
other
changes, such as doing the fdatasync() less often, may be useful too.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-293899781___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

If you are arguing for "opt-in" rather than "opt-out" by default when a macro 
is not explicitly set, then change the "1" in this line:
`
_periodic_fsync =
  rpmExpandNumeric("%{?_periodic_fsync}%{!?_periodic_fsync:1}");
`
Otherwise set %_periodic_fsync as you wish.

There wasn't any significant speedup using a 1Mb rather than a 32Kb buffer when 
implemented.

If you wish fdatasync to be called every ~1Mb, then add logic to call every 
other trip through the loop by, say, interpreting a small value of 
%_periodic_fsync as a count, and a large value as a desired fdatasync interval, 
and proceed accordingly.

If you are saying that there is no interest in ensuring either that RPM always 
syncs to disk, or that there aren't other systems where package management 
interferes with other running processes, you are incorrect.

The critical factor choosing to do fsync in RPM is going to be knowing the cost 
of using, and only measurements can tell the cost.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-294151888___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

[Jeff Johnson]

Another issue that might make debugedit.c more portable is similar to inlining 
DW_FORM* instead of
`#include `

Instead of using EU version dependent #defines for strtab_init() etc (which 
needs -lebl), it may be more "portable" to just inline the routines and 
maintain a copy in debugedit.c.

That would permit non-elfutils -lelf to be used "portably" even if there is 
code duplication.

I dunno (yet) how much code would need to be duplicated, but will take a look 
soonishly.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/171#issuecomment-284468205___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

[Jeff Johnson]

Yes all the patches were reversed.

Meanwhile I just sorted the first patch with Mark Hatle (aka "fray"). The 
segfault is not reproducible with EU-0.168.

The hex printing was an attempt to avoid pgpHexStr() which differs between 
rpm.org and rpm5.org.

You explained away the valid_file() check

The other changes to debugedit --help are largely cosmetic. I maintain POPT and 
so am slightly OCD ;-)


-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/171#issuecomment-284463573___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Retrofitting a format version on *.rpm packaging. (#172)

[Jeff Johnson]

When originally written, the rpmlead contained information used to identify a 
*.rpm.

Twenty years later, most of the rpmlead information is de facto and ad hoc and 
historical.

Progress using other information, like rpmlib(...) tracking dependencies, has 
managed to associate
new "features" (actually "incompatibilities") with an installer that can handle 
the incompatibilities.
There are also magic numbers associated with each of the 4 components in a 
*.rpm package
(I assume that the new rpmarchive payload has an associated magic, but likely 
doesn't change format version).

(aside)
In fact, a tracking dependency would have avoided the last version 3->4 format 
version change
that caused LSB to dictate that all *.rpm packages MUST be version 3. *shrug*

Tracking dependencies implicitly assume that a Header (with magic) can be 
loaded, and
do not provide a mechanism by which signature/metadata headers can be changed 
to, say,
add a new data type to a Header, which would prevent a header being loaded, 
which prevents
a tracking dependency from being checked.

Fundamental sorts of signature/metadata changes might need format versioning 
(and additional
magic as well as tracking dependencies etc) to be handled correctly, unlike 
changes to a payload
format or payload checksum or adding RichDependencies (to name a few 
incompatibilities that
have been finessed through other means).

I can certainly devise a means to add a new data type to a Header, or replace a 
Header with a
different container structure (*.deb or *.xar sections in files in a wrapper 
archive), or even by
deliberately changing the rpmlead version++, or even by adding a different 
*.rpm suffix naming.

But the core problem remains:
There is nothing but de facto and ad hoc techniques available to change 
*.rpm format.

I think its time to talk about an orderly way to change RPM format rather than 
waiting for
error reports to arrive.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/172___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

[Jeff Johnson]

rpm5.org carries this patch from Yocto:
  https://patchwork.openembedded.org/patch/46887
which likely should be added when updating debug edit.c soon.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/171___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

[Jeff Johnson]

For rpm portability with older versions of elfutils, this harmless sanity check 
might be useful:

(excuse the different hash functions in use in the patch snippet below)
@@ -1533,40 +1493,12 @@
   /* Now format the build ID bits in hex to print out.  */
   {   
 const uint8_t * id = (uint8_t *)build_id->d_buf + build_id_offset;
-char hex[build_id_size * 2 + 1];
-int n = snprintf (hex, 3, "%02" PRIx8, id[0]);
-assert (n == 2);
-for (i = 1; i < (int)build_id_size; ++i)
-  {
-   n = snprintf ([i * 2], 3, "%02" PRIx8, id[i]);
-   assert (n == 2);
-  }
+char *hex = pgpHexStr(id, build_id_size);
 puts (hex);
+free(hex);
   }   
 }
 
-/* It avoided the segment fault while file's bss offset have a large number.
-   See https://bugzilla.redhat.com/show_bug.cgi?id=1019707
-   https://bugzilla.redhat.com/show_bug.cgi?id=1020842 for detail. */
-void valid_file(int fd)
-{
-  Elf *elf = elf_begin (fd, ELF_C_RDWR, NULL);
-  if (elf == NULL)
-  {
-error (1, 0, "elf_begin: %s", elf_errmsg (-1));
-return;
-  }
-
-  elf_flagelf (elf, ELF_C_SET, ELF_F_LAYOUT);
-
-  if (elf_update (elf, ELF_C_WRITE) < 0)
-error (1, 0, "elf_update: %s", elf_errmsg (-1));
-
-  elf_end (elf);
-
-  return;
-}
-
 int
 main (int argc, char *argv[])
 {
@@ -1682,9 +1614,6 @@
   exit (1);
 }

-  /* Make sure the file is valid. */
-  valid_file(fd);
-
   dso = fdopen_dso (fd, file);
   if (dso == NULL)
 exit (1);


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/171#issuecomment-284246637___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

[Jeff Johnson]

Portability of debugedit.c on systems that do not support #include  
(solaris, *BSD) can be achieved by duplicating certain DWARF standard constants:

-#ifndef EM_AARCH64
-#define EM_AARCH64  183 /* ARM AARCH64 */
-#endif
-#ifndef R_AARCH64_ABS32
-#define R_AARCH64_ABS32 258
-#endif 
-
-/* some defines taken from the dwarf standard */
-
-#define DW_TAG_compile_unit0x11
-
-#define DW_AT_name 0x03
-#define DW_AT_stmt_list0x10
-#define DW_AT_comp_dir 0x1b
-
-#define DW_FORM_addr   0x01
-#define DW_FORM_block2 0x03
-#define DW_FORM_block4 0x04
-#define DW_FORM_data2  0x05
-#define DW_FORM_data4  0x06
-#define DW_FORM_data8  0x07
-#define DW_FORM_string 0x08
-#define DW_FORM_block  0x09
-#define DW_FORM_block1 0x0a
-#define DW_FORM_data1  0x0b
-#define DW_FORM_flag   0x0c
-#define DW_FORM_sdata  0x0d
-#define DW_FORM_strp   0x0e
-#define DW_FORM_udata  0x0f
-#define DW_FORM_ref_addr   0x10
-#define DW_FORM_ref1   0x11
-#define DW_FORM_ref2   0x12
-#define DW_FORM_ref4   0x13
-#define DW_FORM_ref8   0x14
-#define DW_FORM_ref_udata  0x15
-#define DW_FORM_indirect   0x16


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/171#issuecomment-284247337___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

[Jeff Johnson]

debugedit.c --help can be improved (and potentially internationalized)

*shrug*

@@ -1323,16 +1290,15 @@

 static struct poptOption optionsTable[] = {
   { "base-dir",  'b', POPT_ARG_STRING, _dir, 0,
-N_("base build DIRECTORY of objects"), N_("DIRECTORY") },
+  "base build directory of objects", NULL },
   { "dest-dir",  'd', POPT_ARG_STRING, _dir, 0,
-N_("DIRECTORY to rewrite base-dir into"), N_("DIRECTORY") },
+  "directory to rewrite base-dir into", NULL },
   { "list-file",  'l', POPT_ARG_STRING, _file, 0,
-N_("FILE where to put list of source and header file names"), N_("FILE") },
+  "file where to put list of source and header file names", NULL },
   { "build-id",  'i', POPT_ARG_NONE, _build_id, 0,
-N_("recompute build-id note and print on stdout"), NULL },
+  "recompute build ID note and print ID on stdout", NULL },
   { "build-id-seed", 's', POPT_ARG_STRING, _id_seed, 0,
-N_("if recomputing the build-id note, use this string as hash SEED"), 
N_("SEED") },
-
+  "if recomputing the build ID note use this string as hash seed", NULL },
 POPT_AUTOHELP
   { NULL, 0, 0, NULL, 0, NULL, NULL }
 };


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/171#issuecomment-284247465___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Print exit code with rpm -vv (#195)

[Jeff Johnson]

This trivial patch helps diagnose rpm behavior while developing or testing.

`
diff --git a/rpmqv.c b/rpmqv.c
index 8655776..4f448c2 100644
--- a/rpmqv.c
+++ b/rpmqv.c
@@ -354,5 +354,7 @@ int main(int argc, char *argv[])
 
 rpmcliFini(optCon);
 
+rpmlog(RPMLOG_DEBUG, "exit code: %d\n", ec);
+
 return RETVAL(ec);
 }
`

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/195___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] ELF file conflict 'color' resolution for tri-lib systems (#193)

[Jeff Johnson]

Resolving file "colors"  affects more than MIPS64 n32: the x32 format also uses 
(1 <<2) color bit and (potentially) needs RPM to choose between executables 
that are compiled differently.

AFAIK, PLD has succeeded in adding support for x32 to RPM5, if you need patches 
that have been tested in the "real world".

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/193#issuecomment-292721953___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

Thanks for measurements! Could you also attempt to measure the effect on RPM 
too? Wall clock time is sufficient, as the time necessary to perform an upgrade 
is what most RPM users are sensitive to.

Your graphs (presumably on SSD?) show smaller/periodic writes w fsync enabled,
and larger/delayed writes w/o fsync, as expected. The delay (most obvious w/o 
sync, but occasionally w fsync(2)) is part of the cache flush algorithm to 
prevent active writes from starving reads (i.e. your database stalls).

The issue (for RPM) is that fwrite(2)+fsync(2) ends with a  blocking system 
call, while mmap(2)+madvise(MADV_DONTNEED)+msync(MS_ASYNC) permits RPM to 
continue running while the cache is flushed and resources are free'd. By 
continuing to run, RPM can provide more information on RPM near term write 
needs to the kernel cache flushing scheduler. The end result in both cases is 
that resources used by RPM write's are free'd as soon as possible.

Does that explanation make sense to you? I'll try to get a patch to (again) use 
mmap+madvise+msync for rpm.org together this weekend.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-292720982___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

The buffer is 32 * BUFSIZ )= 128Kb) not 32Kb. Sorry for the confusion.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-294225532___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

One last note: doing fdatasync once, before fadvise+fsync, not after every 
write is ~10x faster:

`
FDIO:  30 writes,  3763601 total bytes in 0.003558 secs
 FDIO:   1 allocs,  3763601 total bytes in 0.24 secs
 FDIO:   1 dsyncs,0 total bytes in 0.074081 secs
 FDIO:   1  syncs,0 total bytes in 0.005785 secs
`

That makes the best case ~17x slower than using the kernel caches. A cost of 
17x is almost tolerable in RPM for the benefits of having files written to disk 
and not blowing out the kernel caches.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-294217202___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

[Jeff Johnson]

@cgwalters: easy enough to wrap syncfs(2) into a measurement harness:

`
==> Fclose(0x61408e40)  | LIBIO 0x61204fc0(-1) fdno -1 | UFD 16 fp 
0x61204fc0
--> Fincore(0x61408e40) fdno 16 path 
/opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66
<-- Fincore(0x61408e40) fdno 16 rc 919
--> Syncfs(0x61408e40) fdno 16 path 
/opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66
<-- Syncfs(0x61408e40) fdno 16 rc 0
--> Fincore(0x61408e40) fdno 16 path 
/opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66
<-- Fincore(0x61408e40) fdno 16 rc 919
<-- fdClose(0x61408e40) rc 0| UFD -1 fp 0x61204fc0
 FDIO:  30write(s),  3763601 total bytes in 0.003511 secs
 FDIO:  32   digest(s),  3763601 total bytes in 0.034357 secs
 FDIO:   1alloc(s),  3763601 total bytes in 0.40 secs
 FDIO:   1   syncfs(s),0 total bytes in 0.166156 secs
 FDIO:   2   incore(s),  7528448 total bytes in 0.000425 secs
<-- fdClose(0x61408e40) rc fffe | LIBIO 
0x61204fc0(-1) fdno -1 | UFD -1 fp (nil)
D: fini  100600  1 (   0,   0) 3763601 
/opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66
`

For starters, Fincore (which sums the pages reported by mincore(2) on the 
mmap'd fdno) indicates that no pages are removed from the system cache by 
syncfs(2) whatsoever. Likely fdatasync+fadvise are still needed with syncfs(2), 
just like with fsync(2).

Then syncfs(2) is clocking in at ~165 millisecs, which is ~3x worse than 
fdatasync+fadvise+fsync ~50 millisecs. Which is hardly surprising: the ext4 
partition being sync'd is ~500Gb, much larger than the ~4Mb file I am 
measuring. Hard to compare coconuts with bananas ...

(aside)
And sure os-tree can create/use smaller partitions to limit the amount of I/O 
necessary to sync to disk. RPM hasn't that luxury, and adding an artificial 
partition around an I/O benchmark is ... well ... rather a cheat. YMMV.

Finally the wall-clock time for kernel-core RPM install clocks in at

`2.84user 1.04system 3:44.12elapsed 1%CPU (0avgtext+0avgdata 221228maxresident)k
200inputs+75808outputs (3major+92612minor)pagefaults 0swaps`

So syncfs(2) is merely 45x slower than the BASE measurements I reported before.

I'll stick with fdatasync+fadvise+fsync (~30x slower) until I hear something 
better.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/187#issuecomment-295073080___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] RPM+NDB doesn't implement rpmpkgSetFsync() (#288)

[Jeff Johnson]

Changing nab_SetFsync() to do the obvious (but perhaps naive) passing:
```
static void ndb_SetFSync(rpmdb rdb, int enable)
{
struct ndbEnv_s *ndbenv = rdb->db_dbenv;
if (ndbenv->xdb)
rpmxdbSetFsync(ndbenv->xdb, enable);
if (ndbenv->pkgdb)
rpmpkgSetFsync(ndbenv->pkgdb, enable);
}
```

leads to a slight (but still minutes, not seconds) improvement in --rebuilddb 
performance:
```
[jbj@ji rpm]$ /usr/bin/time sudo ./rpmdb --rebuilddb --ndb
3.42user 4.01system 7:34.55elapsed 1%CPU (0avgtext+0avgdata 46952maxresident)k
97664inputs+3641232outputs (0major+520873minor)pagefuls 0swaps
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/288#issuecomment-318846242___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Offer LMDB as an alternative engine to BDB for rpmdb (#128)

[Jeff Johnson]

@leo-yuriev: libmdbx looks very nice, but there are several issues that need to 
be fixed in an rpmdb schema before attempting better back ends. See other 
issues I have reported against RPM ...

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/128#issuecomment-318850555___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] hdrNum should be stored in big-endian, not native endian, for btree retrieval (#287)

[Jeff Johnson]

Question #5 in the Berkeley DB Access Method FAQ says this:

```
I'm using integers as keys for a Btree database, and even though the key/data 
pairs are entered in sorted order, the page-fill factor is low.
This is usually the result of using integer keys on little-endian architectures 
such as the x86. Berkeley DB sorts keys as byte strings, and little-endian 
integers don't sort well when viewed as byte strings. For example, take the 
numbers 254 through 257. Their byte patterns on a little-endian system are:

254 fe 0 0 0
255 ff 0 0 0
256  0 1 0 0
257  1 1 0 0
If you treat them as strings, then they sort badly:

256
257
254
255
On a big-endian system, their byte patterns are:

254 0 0 0 fe
255 0 0 0 ff
256 0 0 1 0
257 0 0 1 1
and so, if you treat them as strings they sort nicely. Which means, if you use 
steadily increasing integers as keys on a big-endian system Berkeley DB behaves 
well and you get compact trees, but on a little-endian system Berkeley DB 
produces much less compact trees. To avoid this problem, you may want to 
convert the keys to flat text or big-endian representations, or provide your 
own Btree comparison.
```

RPM+BDB currently uses a hash for secondary lookup and so is largely unaffected 
as distributed.

However LMDB has only btree, and using little endian hdrNum comparison has the 
following consequences:
* the distribution of keys across btree leaf nodes is sub-optimal
* the depth of search is biased, causing btree performance degradation because 
of tall/skinny trees
* the ability of btree to always preserve keys in sorted order is lost. E.g. an 
iteration across Packages does not return packages in the order installed even 
though the hdrNum is most definitely assigned as a monotonically increasing 
integer.

Both BDB and LMDB permit adding a custom btree comparison that can do the 
comparison correctly.
The problem with customizing the comparison (instead of always using big endian 
hdrNum) is that
the custom comparison MUST be configured for all accesses. This effectively 
makes mdb_dump/mdb_load as distributed with LMDB useless (the tools could of 
course be patched to add a custom comparison, but there's no easy way to force 
only the modified tools to be used in the real world).

I will send a patch to fix the issue shortly. Making the effect of the patch 
invisible to users is all that is hard.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/287___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] RPM backend performance is limited by arrays of hdrNum's (#290)

[Jeff Johnson]

The following callgraphs for BDB/LMDB/NDB all show a common hotspot retrieving 
arrays of hdrNum's from indices.

The performance problem shows up worst on add/del operations, where a RMW loop 
has to be performed to add/del a hdrNum item to an array. The array is then 
sorted (and perhaps uniqified) by qsort(3) repeatedly, the worst case behavior 
for the algorithm, resorting almost sorted arrays (merge sort or even a home 
rolled insertion loop would be less costly).

Maintaining the hdrNum's endianness is another flaw; exposing the hdrNum's 
through the RPM API is yet another flaw because the values will change with 
every --rebuildb (i.e. the hdrNum's are not persistent).

The fundamental architectural problem that needs solving for better performance 
is the nesting of per-header and then per-tag operations performed by 
rpmdbAdd(). Ideally, a batch mode update for each index of all the headers 
would remove the need to constantly reread/modify/rewrite.

One approach to removing the overhead associated with the array management that 
"works" with BerkeleyDB is to tie the secondary index to the primary store 
using db->associate. Then Berkeley DB can handle the caching/optimizations 
needed to handle indices transparently to RPM.

Using db->associate in Berkeley DB is essentially the same as using a SQL 
trigger to maintain indices derived from a primary store, a very common 
abstraction used with RDBM's.

Here are the callgraphs that show the performance bottleneck for all of 
BDB/NDB/LMDB:

BDB
===
```
[jbj@ji rpm]$ /usr/bin/time sudo ./libtool --mode=execute /home/jbj/bin/cg 
./rpmdb --rebuilddb
208.17user 3.31system 3:32.94elapsed 99%CPU (0avgtext+0avgdata 
74000maxresident)k
0inputs+492608outputs (0major+37493minor)pagefuls 0swaps
```
[bdb.cga.gz](https://github.com/rpm-software-management/rpm/files/1185163/bdb.cga.gz)

NDB
===
```
/usr/bin/time sudo ./libtool --mode=execute /home/jbj/bin/cg ./rpmdb 
--rebuilddb --ndb
99.59user 3.67system 8:35.82elapsed 20%CPU (0avgtext+0avgdata 93888maxresident)k
0inputs+3315224outputs (0major+461509minor)pagefuls 0swaps
```
[ndb.cga.gz](https://github.com/rpm-software-management/rpm/files/1185165/ndb.cga.gz)

LMDB

```
[jbj@ji rpm]$ /usr/bin/time sudo ./libtool --mode=execute /home/jbj/bin/cg 
./rpmdb --rebuilddb --lmdb
113.50user 1.57system 1:55.07elapsed 99%CPU (0avgtext+0avgdata 
393692maxresident)k
0inputs+455720outputs (1103major+129040minor)pagefuls 0swaps
```
[lmdb.cga.gz](https://github.com/rpm-software-management/rpm/files/1185166/lmdb.cga.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/290___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] RPM+NDB should implement ndb_dump/ndb_load/ndb_stat for database dumps. (#289)

[Jeff Johnson]

Converting databases is painful, particularly if both forward/backward 
conversions are desired.

BerkeleyDB has a set of utilities to do this conversion, LMDB uses an almost 
identical set of utilities.

While rpmdb --importdb/--exportdb are a high performing KISS  improvement on 
the BDB-style dump/load, there are still needs to display key<->value pairs in 
human readable (i.e. not just hex/binary) format.

The LMDB licensed mdb_dump/mdb_load utilities could be swiped and modified to 
handle NDB pretty easily to provide a well-understood (and used by libvirt 
iirc) common import/export/format.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/289___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] hdrNum should be stored in big-endian, not native endian, for btree retrieval (#287)

[Jeff Johnson]

NDB appears to not deal with database portability between LE and BE machines. 
So I'll leave that patch for NDB to implement. The change from native -> big 
endian is likely a noop anyways because NDB is based on hashed access of hdrNum 
primary keys.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/287#issuecomment-318876672___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Offer LMDB as an alternative engine to BDB for rpmdb (#128)

[Jeff Johnson]

Google Translate works just fine: time to become pluralistic even if you can't 
crack a joke in Russian.

-- 
You are receiving this because you commented.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/128#issuecomment-318869365___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

[Jeff Johnson]

The last patch to RPM_CHECK_LIB is appropriate for "opt-out" builds (i.e. 
--without-lmdb).

Leave "no" if willing to explicitly "opt-in" by passing --with-lmdb to 
configure.

For a serious test in CI, arrange for the macro %_db_backend lmdb to be set by 
default. Almost certainly you will also find that the lock.mdb file will need 
0666 permissions when creating/using a database as non-root. The existing 
implementation will propagate that permission through --rebuilddb if set once 
when a database is created.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/291#issuecomment-318876956___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The attached patch adds allocate/fadvise/fdatasync/fsync functions in rpmio 
with the calls on I/O intensive install code paths.

In addition, two flags have been added to Fopen(): 'S' to do fsync-on-close, 
and '?' to do per-fd debugging.

Simple benchmarks for a kernel package installation (rechecking PR #197 
measurements)

without fsync-on-close:

$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm 
...
5.67user 0.54system 0:07.45elapsed 83%CPU (0avgtext+0avgdata 29780maxresident)k
0inputs+303336outputs (14major+18667minor)pagefuls 0swaps

with fsync-on-close

$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm 
...
5.70user 0.87system 1:15.05elapsed 8%CPU (0avgtext+0avgdata 29772maxresident)k
8inputs+306472outputs (14major+19052minor)pagefuls 0swaps

So fsync-on-close is  ~10x slower on whatever disk I happen to be using on 
CentOS7.

The next step will be to use libeio to run fdatasync+fadvise+fsync+close 
asynchronously.

[rpm+fsync.patch.gz](https://github.com/rpm-software-management/rpm/files/1199322/rpm.fsync.patch.gz)




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320161172___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add missing "il" to header sanity check error message (9a6a95b)

[Jeff Johnson]

How is this failure considered a "tag number mismatch"?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/commit/9a6a95bf0b3fa60f64f531e1c28eb93743caa41b#commitcomment-23660624___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Macro arguments do not support quoting (#222)

[Jeff Johnson]

Leave bad enough alone imho.

Every macro templating language in the world has quoting problems: its the 
nature of the beast.

And a Principle of Least Surprise "Do what bash does!" sets expectations that 
are wildly out of line with reality.

FWIW, the only place that macros MUST be used (instead of bash/lua/etc) is in 
the preamble and in %files. Use whatever language you wish to program in, and 
use macros for tempting, not programming.

RPM macros are a gawd awful way to attempt to solve programming issues ...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/222#issuecomment-322269707___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

[Jeff Johnson]

Note that a package produced by rpm 4.13 reproduced the problem.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/270#issuecomment-322454866___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

[Jeff Johnson]

Sigh ... there are much better fixes for ^C handling than reopening an rpmdb 
(where ^C will *still* be disabled while installing).

For starters, the signal handling in RPM was designed to be polled at code 
points where the rpmdb is known to be consistent (for a graceful exit with an 
rpmdbClose when that makes sense).

Adding the poll in 10-15 places in rpm code SHOULD lead to reasonable ^C 
handling (i.e. responsiveness in seconds, not minutes) as well as no rpmdb 
stateful problems. Run a list
of packages that haven't been successfully upgraded and remove on exit if you 
want to start
towards transactional durability.

There is also an approach using atexit(2), as well as an approach with chained 
signal handlers (but that will likely need collusion with depsolvers that use 
rpmlib).

The entire issue is that SOME application MUST call rpmdbClose as part of ^C 
handling.

Pretending that O_RDONLY doesn't write locks in the dbenv (including the 
transaction lock) when run as root is an act of ignorant denial.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/251#issuecomment-315191663___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Convert output to the current locale (#250)

[Jeff Johnson]

The more modern interface is iconv(3).

Here is the basic algorithm. The right implementation adds what is known as a 
header format extension to convert the encoding of any output displayed by rpm 
--query (aka headerSprintf() ...).

hth

[iconv.code.gz](https://github.com/rpm-software-management/rpm/files/1146620/iconv.code.gz)




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/250#issuecomment-315197780___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

[Jeff Johnson]

@mischroe: DB_PRIVATE has its own associated durability/signal problems, 
including that ^C is masked while installing, and with the transaction lock 
being for exclusive writers, not readers.

Truly ^C handling needs to be solved intelligently, not with convoluted 
contortions to BDB configuration. Been there, done that, hacks to BDB 
configuration and verification ever worked well enough. Adding ~10-15 checks on 
signal receipt and calling rpmdbClose exits within seconds.

The entire issue is that rpmdbClose needs to be called before exit, 
particularly with ^C.

And most of the complaints about "corruption" (for the entirely different case 
of "kill -9") disappear if 10 lines of code are added to automate 
DB_RUNRECOVERY.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/251#issuecomment-315358868___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

[Jeff Johnson]

I'm happy it works fine for SuSE: I'm sure you have tested.

Meanwhile ^C exiting and calling rpmdbClose is a problem that needs to be 
solved without convoluted BDB configuration changes. Its tiresome to 
continuously revisit this issue, in patches and bug reports.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/251#issuecomment-315361310___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

[Jeff Johnson]

The RPM_CHECK_LIB macro: add to acinclude.m4.
[rpm_check_lib.m4.gz](https://github.com/rpm-software-management/rpm/files/1156106/rpm_check_lib.m4.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/257#issuecomment-316076403___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

[Jeff Johnson]

There are (at least) 3 new FSE compression formats that have been released in
the past few years: zstd (Facebook),  brotli (Google), lzfse (Apple).

The new compressors have better compression ratios and speed than gzip/zlib.

And while then compression rations is still not as good as XZ/LZMA, the 
decompression speed (which is mostly rate determining for RPM installation 
speed), of zest is considerably faster than XZ/LZMA.

One or more of these implementations might be useful for RPM.

(aside)
There's an implementation for RPM+ZSTD available here: 
[http://rpm5.org/files/rpm/rpm-5.4/SNAPSHOT/rpm-5.4.18-0.20170718.src.rpm](url) 
that can be adapted to RPM4. I can/will do the patch if there is sufficient 
interest.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/256___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

[Jeff Johnson]

Here are URL's for zstd, brotli, and lzfse"
- [https://github.com/facebook/zstd.git](url)
- [https://github.com/google/brotli.git](url)
- [https://github.com/lzfse/lzfse.git](url)



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/256#issuecomment-316037161___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

[Jeff Johnson]

Projects that use autoconf are expected to detect build prerequisites and 
libraries.

This usually leads to a set of ad hoc de facto m4 macros that deal with various 
details like API and path incompatibilities, much of which is platform and os 
and distro dependent.

Various standard tools (like pkgconfig) to find these details have been 
devised; meanwhile there is still a great deal of diversity, particularly 
between platforms like linux and *BSD, that often makes portability more 
difficult than it needs to be.

One useful approach -- particularly when maintaining a project like RPM -- is 
RPM_CHECK_LIB that does all the usual operations and permits a compact 
specification of 3rd part library details.

Here is the blurb from the RPM_CHECK_LIB m4 macro (I will send along the 
complete addition to acinclude.m4 later today):

```
dnl ##
dnl ##  NAME:
dnl ##RPM_CHECK_LIB -- Check for third-party libraries
dnl ##
dnl ##  COPYRIGHT
dnl ##Copyright (c) 2007 Ralf S. Engelschall 
dnl ##
dnl ##  DESCRIPTION:
dnl ##This is a rather complex Autoconf macro for sophisticated
dnl ##checking the availability of third-party libraries and
dnl ##extending the build environment for correctly building
dnl ##against it.
dnl ##
dnl ##It especially supports the following particular features:
dnl ##- is aware of old-style [lib]-config style scripts
dnl ##- is aware of new-style pkg-config(1) [lib].pc configuration 
files
dnl ##- searches under standard sub-directories "include", "lib", etc.
dnl ##- searches under arbitrary sub-areas of a tree like ".libs", etc.
dnl ##- searches in standard system locations (implicitly)
dnl ##- supports searching for function in multiple libraries
dnl ##- supports searching for multiple headers
dnl ##- supports multiple search locations (fallbacks!)
dnl ##
dnl ##  USAGE:
dnl ##  - configure.in:
dnl ##RPM_CHECK_LIB(
dnl ##,-- [$1] e.g. GNU bzip2
dnl ##, -- [$2] e.g. bzip2
dnl ##,-- [$3] e.g. bz2
dnl ##,-- [$4] e.g. BZ2_bzlibVersion
dnl ##,  -- [$5] e.g. bzlib.h
dnl ##[,-- [$6] e.g. yes,external:internal:none
dnl ##[, -- [$7] e.g. lib/bzip2:include:src
dnl ##[,  -- [$8] e.g. AC_DEFINE(USE_BZIP2, 1, 
[...])
dnl ##-- [$9] e.g. AC_MSG_ERROR([...])
dnl ##]]])
dnl ##
dnl ##  - Makefile.in:
dnl ##top_srcdir = @top_srcdir@
dnl ##srcdir = @srcdir@
dnl ##WITH_= @WITH_@
dnl ##WITH__SUBDIR = @WITH__SUBDIR@
dnl ##CPPFLAGS   = @CPPFLAGS@
dnl ##CFLAGS = @CFLAGS@
dnl ##LDFLAGS= @LDFLAGS@
dnl ##LIBS   = @LIBS@
dnl ##
dnl ##  - CLI:
dnl ##$ ./configure \
dnl ##  --with-[=]
dnl ##  [...]
dnl ##
dnl ##  SYNTAX:
dnl ##   ::=  |   
dnl ##   ::=  "," 
dnl ##  ::= "yes" | "no"
dnl ##  ::=  ":" 
dnl ##   | 
dnl ##   | "system"
dnl ##   | "external"
dnl ##   | "internal"
dnl ##   | "none"
dnl ## ::= [...] /* valid arg for test(1) option "-d" */
dnl ##
dnl ##
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/257___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

[Jeff Johnson]

Here is a moderately complex real world example of how to use: detecting ZSTD 
either externally or internally.

In configure.ac:
```
dnl # Facebook Zstd
RPM_CHECK_LIB(
[Facebook Zstd], [zstd],
[zstd], [ZSTD_versionNumber], [zstd.h],
[no,external:internal:none], [zstd:lib:lib],
[
], [])
```

When the generated configure is invoked with --with-zstd=internal (with a 
github zstd git checkout in the top level directory) succeeds, then these 
values are available in Makefile's for adding to the usual libtool variables:

```
WITH_ZSTD = yes
WITH_ZSTD_CPPFLAGS =  -I$(top_srcdir)/$(WITH_ZSTD_SUBDIR)/lib 
-I$(top_builddir)/$(WITH_ZSTD_SUBDIR)/lib -I$(top_srcdir)/$(WITH_ZSTD_SUBDIR) 
-I$(top_builddir)/$(WITH_ZSTD_SUBDIR)
WITH_ZSTD_LDFLAGS =  -L$(top_builddir)/$(WITH_ZSTD_SUBDIR)/lib 
-L$(top_builddir)/$(WITH_ZSTD_SUBDIR)
WITH_ZSTD_LIBS =  -lzstd
WITH_ZSTD_SUBDIR = zstd
```

The equivalent Makefile values for configure --with-zstd=external success:
```
WITH_ZSTD = yes
WITH_ZSTD_CPPFLAGS =
WITH_ZSTD_LDFLAGS =
WITH_ZSTD_LIBS =
WITH_ZSTD_SUBDIR =
...
LIBS = ... -lzstd
```


And config.h of course contains
```
/* Define as 1 if building with Facebook Zstd library */
#define WITH_ZSTD 1
...
/* Define to 1 if you have the `zstd' library (-lzstd). */
#define HAVE_LIBZSTD 1
...
/* Define to 1 if you have the  header file. */
#define HAVE_ZSTD_H 1
```

Note: I've left out the definition's for HAVE_LIBZSTD and HAVE_ZSTD_H when 
using --with-zstd=internal for simplicity of explanation. In practice, these 
definitions are seldom needed with an internal zest source tree.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/257#issuecomment-316087315___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

[Jeff Johnson]

The often stated public policy here with RPM4 is "Compatibility with RPM5 is a 
negative goal."
Don't hold your breath for a patch: I will generate and distribute a patch as 
soon as I understand
the full consequences of recent changes to the RPM format. 

Yes, Yocto issues are well known. The rpm5+dnf incompatibilities were grossly 
exaggerated.
*shrug* whatever.

FYI: I maintain RPM5, and have several (like almost 10 ;-) years of experience 
dealing with Poky/Yocto packaging issues.

Meanwhile I suspect that capsule is only the first, not the last, RPM5 package 
that you will see. Ripping out the "exact_size" test will solve the issue until 
there is more info or a better fix available.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/270#issuecomment-316903692___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

[Jeff Johnson]

That is a package produced by RPM5:
```
$ /opt/local/bin/rpm -qp --yaml ./capsule-01.00.12-r0.core2_64.rpm | grep 
Rpmversion
  Rpmversion: 5.4.16
```

The package uses a non-repudiable signature and verifies fine:

```
$ /opt/local/bin/rpm -qvvp  ./capsule-01.00.12-r0.core2_64.rpm
...
D:   PUB: 3375C959 31F7927C V4 DSA
D: == DSA pubkey id 3375c959 31f7927c (package)
D: ./capsule-01.00.12-r0.core2_64.rpm: Header V4 DSA signature: OK, key ID 
31f7927c
```

However, RPM4 can't find the pubkey contained in the package.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/270#issuecomment-31657___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

[Jeff Johnson]

btw, this command is sufficient to identify the rpm version that built the 
package:
`rpm -qp --qf '%{rpm version}\n' intel-aero-repo-1.4-r0.corei7_64.rpm`

It looks the  arguments passed used in the exact_size check are incorrect (the 
"ril" and "rdl" values in the error message seem wrong).

Commenting out this code permits reading the package:
```
--- a/lib/header.c
+++ b/lib/header.c
@@ -1840,7 +1840,9 @@ static rpmRC hdrblobVerifyRegion(rpmTagVal regionTag, int 
rasprintf(buf,
_("region %d: tag number mismatch %d ril %d dl %d rdl %d\n"),
regionTag, blob->il, blob->ril, blob->dl, blob->rdl);
+#ifdef HACK
goto exit;
+#endif
 }
 
 blob->regionTag = regionTag;
```

The real fix is somewhere else, in the value passed to the exact_size boolean 
argument, but will have the same effect of not checking the index/data length 
of a signature header immutable region.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/270#issuecomment-317180902___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

[Jeff Johnson]

The RPM+UUID patch in issue #269 illustrates another, more subtle usage feature 
of RPM_CHECK_LIB.

There are two uuid.h files on a linux system: one from ossp-uuid, the other 
from ext4.

Using --with-uuid=system (rather than --with-uuid=external) finds the correct 
uuid.h.

Think about the hack-o-rounds that usually get implemented in AutoFU to handle 
annoying detection weirdnesses like a file name collision ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/257#issuecomment-317181712___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

[Jeff Johnson]

Specific to OS X portability: there are no efforts to port RPM to OS X (or Mac 
OS) natively (i.e. using only utilities that come from the OS vendor) that I am 
aware of for ~15 years. All existing porting efforts involve Fink, MacPorts or 
Homebrew, and so "portability" includes duplicating existing system 
functionality:

> Use the PATH, Luke.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/275#issuecomment-317477965___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

[Jeff Johnson]

Again: my 1st suggestion was for a removal from git to avoid an annoying 
warning when the file (as always) is replaced by autoreconf.

I added historical context about why an otherwise generated file was added to 
CVS (at the time),
and have personally seen the annoyances that happen when otherwise generated 
boilerplate
ends up under source code control.

At the time I was also personally involved in maintaining RPM on HPUX and 
Irirx, on systems that were using modems for internet connectivity, and so I 
also added /usr/lib/rpm/mkinstalldirs to avoid the tedious task of fetching a 
copy of mkinstalldirs.

This was *NOT* a suggestion that
1) mkdir is portable so that mkinstalldirs can be removed everywhere.
2) mkinstalldirs should not be distributed in tarballs
3) there aren't portability problems between linux and other operating systems: 
both ways.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/275#issuecomment-317474089___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

[Jeff Johnson]

FWIW, I have *repeatedly* argued for native (i.e. --prefix=/usr, etc etc) 
installation on OS X to no avail.

I have also successfully ported and run RPM on OS X for many years.

Meanwhile I return to my original statement:
> ... there are no efforts to port RPM to OS X (or Mac OS) natively ...

by vote of the hoi polloi lemmings.

Meanwhile getting mkinstalldirs out of RPM git hurts nothing.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/275#issuecomment-317607754___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] add RPM_CHECK_LIB m4 macro (from jbj) (#282)

[Jeff Johnson]

Only acinclude.m4 is needed AFAIK.

Apologies for the 2 patches at issue #257: I forgot about the handful of other 
simple macros that RPM_CHECK_LIB needs/uses when I first posted.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/282#issuecomment-318351437___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

[Jeff Johnson]

Next plateau: Convert a BDB -> LMDB database.

Two simple scripts to assist with the conversion.

[bdbexport.sh.gz](https://github.com/rpm-software-management/rpm/files/1180126/bdbexport.sh.gz)

[lmdbimport.sh.gz](https://github.com/rpm-software-management/rpm/files/1180129/lmdbimport.sh.gz)

Note that you will need mdb_dump/mdb_load/mdb_stat executables, which seem not 
to be included in the CentOS7/Fedorable lmdb packages for whatever reason.

The only subtlety -- and the same issue exists with RPM+BDB, but unseen because 
hash rather than btree is used -- in the conversion is byte swabbing the 
primary keys in Packages.

And finally, here is the patch to RPM and the lib/backend/lmdb.c file:

[rpm+lmdb.patch.gz](https://github.com/rpm-software-management/rpm/files/1180172/rpm.lmdb.patch.gz)

[lmdb.c.gz](https://github.com/rpm-software-management/rpm/files/1180173/lmdb.c.gz)

The LMDB implementation has been deliberately left heavily insturmented with 
debugging in order to diagnose issues. There's more that could be done to 
simplify the code: I have deliberately chosen a rather simplistic/redundant 
style while studying what return codes need to be handled.

Assuming there is any interest in RPM+LMDB: judging from the time scale of NDB 
adoption, I am expecting a merge in 2019, and perhaps a discussion of adoption 
in Fedora in 2020, and perhaps a deployment in Fedora 40.

We shall see ...

... but I'm done with the RPM+LMDB issue for now. Have fun!



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/281#issuecomment-318373087___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

[Jeff Johnson]

There have been years of vitriol about Berkeley DB and numerous suggested 
database replacements for RPM.

LMDB is perhaps the best choice because LMDB is most similar to Berkeley DB in 
API and design.

Attached is a patch to add an LMDB backend to RPM.

I will supply an updated lmdb.c implementation as certain plateau's are 
achieved.

The first plateau is the ability to iterate through all packages retrieving 
headers: DONE.

[rpm+lmdb-20170726.patch.gz](https://github.com/rpm-software-management/rpm/files/1176900/rpm.lmdb-20170726.patch.gz)

[lmdb.c-20170726.gz](https://github.com/rpm-software-management/rpm/files/1176907/lmdb.c-20170726.gz)








-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/281___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

[Jeff Johnson]

One has to factor in the context of "choice": I'm tired of years of belly 
aching about RPM databases from the uninformed. That's a very different context 
than "most similar".

The flaws in the rpmdb implementation have nothing whatsoever to do with what 
database implementation is chosen.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/281#issuecomment-318104268___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

[Jeff Johnson]

Next plateau: Retrieve header from secondary -> primary lookup: DONE.

(stoopid brain fart from rescoping the byte swapped hdrNum with an address in a 
MDB_val)

[lmdb.c-20170726.1.gz](https://github.com/rpm-software-management/rpm/files/1177633/lmdb.c-20170726.1.gz)

Now for package management queries and installations ... then start with 
configuration/conversion dreariness.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/281#issuecomment-318135928___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

[Jeff Johnson]

Next plateau: Ensure --query/--install/--rebuilddb are functional: DONE

(there are fundamental design flaws (and bugs) with the current RPM+BDB: there 
is no concept of "durability" as in ACID that can be discussed meaningfully for 
starters.)

On to conversion: there's almost no reason to deal with configuration (but I'll 
twiddle up some knobs for the clueless to benchmark and feel importantly 
impotent ...

[lmdb.c-20170726.2.gz](https://github.com/rpm-software-management/rpm/files/1178076/lmdb.c-20170726.2.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/281#issuecomment-318184261___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

[Jeff Johnson]

A (lightly tested) RPM+ZSTD patch. You will need RPM_CHECK_LIB() from the 
acinclude.m4 comment #2 on issue #255.

[rpm_zstd.patch.gz](https://github.com/rpm-software-management/rpm/files/1160163/rpm_zstd.patch.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/256#issuecomment-316486887___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

[Jeff Johnson]

Hmmm ... its a reasonable guess that failures on BE platforms might be related 
to endianness, but damfino how endianness becomes an issue on an octet oriented 
I/O API thru python bindings. I's suggest looking closely at the return codes 
from *seek/*tell which have never been well supported in rpmio, no matter what 
is being pretended these days.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/259#issuecomment-316567959___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

[Jeff Johnson]

Either set _romiodebug = -1, or (even better) fish out the change to set the 
FD_t specific debugging flag in cvtfmode() if Fopen(path, "r?.fpio") is the 
invocation. There's really no other way to set FD_t specific debugging because 
FD_t is totally opaque in rpmio/rpmio.c

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/259#issuecomment-316568819___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

[Jeff Johnson]

Hmmm ... endianness *does* become an issue when a int64 is somehow returned 
through an int32: LE will Just Work (for small offers) and BE will fail through 
truncation.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/259#issuecomment-316572500___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

[Jeff Johnson]

You might just as easily drop mkinstalldirs. The original motivation for 
including mkinstalldirs was from last century: certain ancient SysV non-linux 
platforms lacked "mkdir -p". Carrying mkinstalldirs was the solution.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/275#issuecomment-317380887___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

[Jeff Johnson]

UUID's provide a common format for identification and database retrieval.

The attached patch adds a header tag format to RPM queries.

- UUIDv1 time stamps (for events like build/install times):
```
$ ./rpm -q --qf '%{buildtime:uuidv1}\n' bash
c60fd500-bc0a-11e7-804e-003048b801de
```
- UUIDv3 namespace identifiers based on MD5 (like package/header digests)
```
$ ./rpm -q --qf '%{sigmd5:uuidv3}\n' bash
c24fd63e-14c4-32d1-84f5-168a1f2908db
```
- UUIDv4 random nonces (overkill because random, but added for completeness)
```
$ ./rpm -q --qf '%{sigmd5:uuidv4}\n' bash
b7a15d96-c4cd-4e41-8598-6463375cb39f
```
- UUIDv5 namespace identifiers based on SHA1 (like package/header digests)
```
$ ./rpm -q --qf '%{sigmd5:uuidv5}\n' bash
9292a557-f445-5f36-9b79-8e79a6efaaa2
```

The UUIDv3/UUIDv5 name spaces can be configured through optional macros 
(defaults below)
```
%_uuid_authhttp://rpm.org
%_uuid_path/packages
```

For reference, the actual text used for, say, Sha1header, in a namespace UUID 
looks like
```
http://rpm.org/packages/Sha1header/
```
Essentially a prefix (to make the namespace unique) of %_uuid_auth and 
%_uuid_path followed by a tag name and a tag value.

UUID's will be the starting point for a RPM+LMDB implementation used as header 
retrieval keys.

You will need the acinclude.m4 file from issue #257 to use the patch below. 
Adding the hires timestamps from issue #197 would improve the UUIDv1 
granularity (not implemented).



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/269___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

[Jeff Johnson]

[rpm_uuid.patch.gz](https://github.com/rpm-software-management/rpm/files/1163437/rpm_uuid.patch.gz)


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/269#issuecomment-316801741___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

[Jeff Johnson]

There are 3 issues:
1) mkdir -p isn't portable and so mkinstalldirs is needed
2) mkinstalldirs is distributed in /usr/lib/rpm
3) mkinstalldirs is copied and tracked in git or other VCS

My comment was wrto issue #3: mkinstalldirs was originally added to RPM in 
order to assuage the bosses reluctance to use autoconf etc.

I suspect that /usr/lib/rpm/mkinstalldirs is seldom used, and there are 
zillions of copies for those platforms that might need: distributing everywhere 
to please a few oddball platforms (OS/2? Windoize? OS X?)

And the correct fix for racy mkdir breaking parallel builds (as described in 
the autoconf doc cited in pull #205) is to go fix GNU mkdir like install -d (if 
not already done long ago). 

*shrug* removing mkinstalldirs from git changes almost nothing and does fix 
your annoyance with autoreconf modifications.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/275#issuecomment-317393813___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

[Jeff Johnson]

Other nitpicks  in the geeky error message: the check has nothing to do with a 
"tag number mismatch", and lacks an "il" identifier that might have helped 
diagnose the problem slightly more rapidly.

OTOH, I'm not sure anyone can diagnose a problem in the binary header format 
without a reproducer and studying the code.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/270#issuecomment-317182655___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Check for 'gettext' during configure script (#245)

[Jeff Johnson]

Did you build with --disable-nls? There is no way for RPM configure to guess 
whether uninstalled get text is bug of a feature: YMMV, everyone's does.

```
 ./configure --help | grep nls
  --disable-nls   do not use Native Language Support
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/245#issuecomment-312411263___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Adding namespaces for macros (#246)

[Jeff Johnson]

Macro templating/configuration in RPM has become MUST'VE for building.

Meanwhile, there are no conventions for sorting out the definitions, and the 
issues related to loading all macros all the time are getting unmanageable.

E.g. here are the macros defined on a relatively recent Fedora box:

```
$ rpm --version
RPM version 4.13.0.1
$ rpm --showrc | grep '^-' | wc -l
813
```

Almost all of those macros are useful for rpm build templatting and useless 
while installing.

(aside)
In fact, many of those rpm build macros (and other macros) can possibly be 
invoked while installing, with possibly malicious effects, if the paths used to 
load macros are writable (or %{load:...} is expanded within rpm configuration 
or a package scriptlet expansion). This is rather too large an attack surface 
to audit IMHO, YMMV.

One KISS approach to limiting what macros are loaded is to identify which 
macros are needed for installation (e.g. %_dbpath), and which macros are useful 
for rpm build's (but not for installation), and load only what is necessary.

A more general approach (quite obvious: many languages have added namespaces to 
rationalize de facto organically grown global definitions) would be to invent 
an explicit syntax to limit the scope of macro loading and expansions that 
includes more than just the RPM  build vs install context.

RPM macros already have a "rpmmc" context container that was intended for 
namespacing (but never used beyond differentiating the global context from the 
RPM CLI override context). 

Adding an explicit namespace syntax for an "opt-in" implementation is not much 
harder than choosing a syntax. Since macro names are already limited (i.e. must 
be >2 characters, must be alphabetic+numeric+underscore, etc), choosing a 
namespace separator is largely a choice of a parseable token. E.g. it wouldn't 
be impossibly hard to hack in a C++-like namespace identifier based on the 
token '::' without breaking the existing usage of ':' in the parser. There are 
surely many other unused ASCII characters that could be substituted for '::'

Another implicit approach that seems viable would be to overload an embedded 
'_' character within a macro name as if it were a namespace: that "works" 
fairly well with, say, per-interpreter macro namespaces like "perl" and 
"python" and "ruby", but would require some magic filtering for, say, 
self-describing macro names that were created ad hoc like

```
-14: _binaries_in_noarch_packages_terminate_build   1
-14: _empty_manifest_terminate_build1
-14: _invalid_encoding_terminates_build 0
-14: _missing_build_ids_terminate_build 1
-14: _missing_doc_files_terminate_build 1
-14: _python_bytecompile_errors_terminate_build 1
-14: _unpackaged_files_terminate_build  1
-14: _wrong_version_format_terminate_build  1
```

OTOH RPM is certainly well positioned to be able to identify its own 
configuration, and map into a namespace.

I dunno what the best approach is: all the usual "opt-in" vs "opt-out" 
approaches, as well as "per-vendor syntax du jour", are viable approaches.

Meanwhile I suggest its time to find a convention that "works" for mapping 
macros into namespaces/files.







-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/246___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Pre-validate (all) header data for semantic consistency (#242)

[Jeff Johnson]

As long as --nosignature and --nodigest are commonly used everywhere -- 
including in depsolvers -- you can setup an internal table to associate the 
data type with the tag and never access the field in the header.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/242#issuecomment-311616024___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The attached patch does a rude-and-crude libeio pipe driven event loop to ramie.

The patch adds an 'N' (mnemonic: O_NONBLOCK) flag to Fopen() flags.

The libeio loop is lazily started, and atexit(3) is used to wait for all events 
to be completed.

The operations of fdatasync/fadvise/fsync are pipelined (i.e. fdatasync/fsync 
are each separate events).

Basically libeio is being used as a co-routine monitor to move the costly 
operations off  the main thread of operation in order to do a wall clock 
benchmark:

```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.67user 0.88system 0:39.75elapsed 16%CPU (0avgtext+0avgdata 32420maxresident)k
0inputs+22outputs (0major+13681minor)pagefuls 0swaps

```

So ~5x slower than no fsync(3) which is currently implemented in RPM.

Which is starting to get to a reasonable execution cost for the simultaneous 
benefits of
* all file operations are durable (in the sense that all installed files are 
known written to disk).
* all dirty pages in the buffer cache are invalidated (so there is no I/O cache 
blowout from performing an RPM upgrade).

The underlying goal here was to assess whether a libeio event loop (which is 
far more general than fdatasync/fsync operations here) has benefits if 
implemented in RPM. The rude-and-crude implementation here is already 2x faster 
than the synchronous rsync-on-close implementation, and is not much more 
complex. JMHO, YMMV, everyone's does.

[rpm+fsync+libeio.patch.gz](https://github.com/rpm-software-management/rpm/files/1202849/rpm.fsync.libeio.patch.gz)

The next step(s) will be to do a proper RPM+LIBEIO implementation so that 
libeio can be used everywhere as needed, not just on the peculiar 
fire-and-forget fsync-on-close file write path used here.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320484006___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

There is one other alternative: libcoro (also by Marc Lehman). If the only 
problem I was trying to solve was making fdatasync+fsync asynchronous, then 
libcoro would be the best choice. My patch essentially reduces libeio to 
co-routine functionality. libcoro does not require a scheduler, nor threads, 
etc if there is a non-blocking variant of an existing system call that needs to 
be run.

Meanwhile there are serious design issues switching to a coroutine 
implementation, the code is often quite mysterious.

There are similar issues reworking the design of existing imperative I/O loops 
to run asynchronously. However, the issues that need to be dealt with in an 
event loop are rather easier than redesigning with coroutines.

Again, JMHO, YMMV, everyone's does. libeio is what I've chosen to attempt an 
event loop implementation in RPM, try-and-see-and-measure is the only 
development paradigm I believe in ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320491272___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

Yes, there is libevent, then libev+libeio, and now libuv, all tuned towards 
slightly different goals.

libeio is narrowly targeted at asynchronous system calls, can be embedded, and 
is generally lean-and-mean and well tested.

But go shopping for alternatives if you wish: I chose libeio after looking at 
multiple alternatives.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489014___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The main reason why libevent was not used:

* Marc Lehmann  chose to move on from libevent -> 
libev+libeio and strip out complex buffer handling. Small and targeted and 
embeddable are all KISS; automating buffer allocation handling is hardly useful 
or necessary to rpmio.

The main reason why libuv (and nodejs) are not appropriate for RPM:
* libuv/nodejs are about asynchronous networking: nothing much within RPM4 I/O 
needs/uses any network related system calls, so most of libuv is overkill.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489822___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The only sane way to add an event loop within RPM is through embedding, at 
which point maintenance is with RPM, not through actively maintained external 
libraries. libeio is the only reasonable embeddable choice.

Extending external libeio with additional system calls is trickier with 
external libeio as well because libeio includes both high level wrappings as 
well as a low level core engine. Again embedding is easier to extend and 
straightforward to maintain (if extending libeio to additional system calls 
like posix_fadvise, or David Howell's new-fangled kernel xstat system call). 
Meanwhile what is on libeio suffices for most usage cases, now and for the 
future, imho.

Do not fool yourself thinking otherwise: meanwhile libelo is mature (i.e slowly 
changing) and well tested, and Marc Lehman responds quickly to questions (also 
tested by me ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320490349___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

Some minor tinkering on the fdatasync -> fsync pipeline with code like this:
```
pthread_yield();
if (eio_npending()) rc = eio_poll();
```
to minimize the number of calls to eio_poll() (which always processes at least 
one request) gives these timings

```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.67user 0.74system 0:17.65elapsed 36%CPU (0avgtext+0avgdata 29880maxresident)k
0inputs+300120outputs (0major+19150minor)pagefuls 0swaps
```

So using an event loop for asynchronous fsync-on-close is ~2.5x slower than not 
doing fsync, and has the added benefit)s) of filesystem durability and no I/O 
cache blowout on large upgrades.

That is a quite acceptable overhead for always doing fsync-on-close IMHO: YMMV 
as always.

Note that no scripts are running in my silly 1 kernel upgrade testing: 
overlapping fsync-on-close with scriptlet execution may result in further 
improvements in The Real World.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320493837___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

And one more try at speeding up fsync-on-close by adding more threads that can 
block:
```
eio_set_min_parallel(512);
eio_set_max_idle(512);
```

With 512 (the default is 4) backend threads that can block, the kernel install 
time is
```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.77user 1.10system 0:17.79elapsed 38%CPU (0avgtext+0avgdata 37968maxresident)k
0inputs+300120outputs (0major+21579minor)pagefuls 0swaps
```

And both fdatasync+fsync are being run (by inspection).

So the reasoning in the previous comment (~2.5x slower for fsync-on-close, 
quite acceptable, etc) seems correct correct even if I had a brain fart while 
tinkering ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320499489___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint