Re: [Rpm-maint] relicensing for Maximum RPM

of the ones who worked for rht at the time they were writing it. So I suspect Paul and jbj's contributions are (c) Red Hat inc. But Matthias and Ville are (c) themselves, unless otherwise disclaimed. You may suspect all you wish, but you better check dates before making false claims in

[Rpm-maint] Keep my name out of your discussions

Re: BTW, Jeff J. must have changed his mind at some point, because rpm5 allows short-circuiting of -bb. You have no right to discuss the state of my mind. In fact, you haven't a clue. Keep my name out of your gossipy garbage. 73 de Jeff ___

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

My bad for not noticing. Good that padding has been added by adding (apparently) 0x1000 to the data size: `0060: 8ead e801 0006 1074 ...t` Adding a "free space" tag (as @rpm5.org) provides an indication that the signature header is padded. Meanwhile, the

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

Ah, you have added RPMSIGTAG_RESERVEDSPACE already which I was not expecting. The (minor but annoying problems) with the assignment to 1008 is that RPMSIGTAG_RESERVEDSPACE collides with RPMTAG_INSTALLTIME, and will not sort correctly to the end of an immutable region data section as a "free

[Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

The recently added RPMTAG_PAYLOADDIGEST computes a digest on the compressed payload. Perhaps now is the time to change the plaintext used to be the uncompressed rather than the compressed archive payload? The benefit of the digest on the uncompressed archive is that the digest becomes immune

[Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

The signature header contains (optional) information to verify package integrity. So new tags are added when packages are signed, and a new signature header is created and inserted into a *.rpm package file. However, the insertion forces a rewrite of the metadata header and payload, which for

[Rpm-maint] [rpm-software-management/rpm] Adding/moving RPMTAG_PUBKEYS to the signature header (#182)

RPM package files can include public keys in the metadata header. So in principle, a package can include the public key used to sign a package and verify the package signature when reading package headers. The RPMTAG_PUBKEYS array is also within the signed immutable region and cannot be

[Rpm-maint] [rpm-software-management/rpm] Tags in the signature header are not sorted (#181)

This is the hex dump of the createrepo_c Archer-3.4.5-6.x86_64.rpm signature header. RPMSIGTAG_SHA1 follows RPMSIGTAG_MD5 at offset 0xa0, forcing a sort while loading the signature header. _ 0060: 8ead e801 0005 0054 ...T 0070: 003e 0007

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Do you have measurements of the effect of fsync()? The last time sync/fdatasync were suggested for RPM that I am aware of was ~2011 in Mandriva. The issue comes up every few years this century ... Here was the effect of adding fsync/fdatasync to RPM, measure on otherwise identical system with

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Good if doing in fsm.h ( I haven't looked). Meanwhile a 50x slower installation is intolerable for a package manager even if opt-in. Nor can an application like RPM be expected to detect whether the underlying filesystem is on SSD or USB or rotational disk and adapt accordingly: there simply

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

FYI: several notes (after looking at the patch) 1) The measurements I cited above are using mmap(2) to decompress directly into pages (avoiding a buffer copy) marked with MADV_DONTNEED (which permits delayed/larger writes) to avoid a buffer copy. In addition, there is a check on file size of

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

For completeness (and humor ;-), there is also O_DIRECT. See [https://lkml.org/lkml/2007/1/10/233](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

There is a modestly obscure means to attach *_DONTNEED (to achieve kernel delayed writes and removing pages when I/O is complete) using the existing rpm.org rpmfiArchiveReadToFilePsm() Fwrite->write(3) loop calling setvbuf(3) immediately after Fopen on the fopencookie(3) FILE *. (aside)

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

There is an alternative (but more complicated) lazy implementation that might be attempted by doing a fsync in Fwrite (or fdWrite) when the accumulated I/O passes a certain mark. H ... A general solution to add a resource cap on all rpmio I/O could also be attempted (but isn't easy).

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Hmmm ... glibc (which uses mmap(2) for stdio if requested) would benefit from being able to mark the stdio buffer-in-use with *_DONTNEED using madvise/posix_fadvise. At a minimum, cvtfmode() in rpmio/rpmio.c would need to be extended to pass the glibc-peculier 'm' flag through to fopen(3)

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Another issue is that certain file systems (NFS? DFS? I forget ...) also delay I/O operations until close(2). What I recall is that an application MUST check the return code of close(2) in order to ensure that the I/O was actually performed: it isn't sufficient to check the return code of

Re: [Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

(answering my own question) Short answer: digesting compressed payload instead of compressing the digested payload is expedient. Long answer: RPM goes to some lengths to preserve streaming-without-peeking-at-magic for *.rpm packages. The usual RPM approach is/was to add/save tags in the

[Rpm-maint] [rpm-software-management/rpm] Colorized rpmlog() output (+PATCH) (#198)

SSIA Have fun! [rpm+color.patch.gz](https://github.com/rpm-software-management/rpm/files/915427/rpm.color.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: option for initial-only content in files (#152)

Add * update by patching and make the policies platform and machine specific. Then call the implementation something catchier than ansible or DevOps. Seriously: configuration != package management. -- You are receiving this because you commented. Reply to this email directly or view it on

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

@jayzmh: major invasive change to ... RPM? or something else? I'll happily write the code for rpm.org RPM if you wish. I fail to see how reverting to what RPM used to do (i.e. use mmap with MADV_DONTNEED and most importantly for you using msync(MS_ASYNC) which schedules flush to disk like

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

@jayzmh - add a static variable to expand the macro as a "one shot" ... there are many places in rpm that do that, hurts little (until rpm can capture global configuration object). Meanwhile I recommend the *_DONTNEED mmap implementation instead of fsync. RPM used that for years, and

Re: [Rpm-maint] [rpm-software-management/rpm] Make 'rpm -V' more resistent against rpmdb manipulations (#196)

Hmmm ... its not clear what exploit is used (from just reading the file at the URL you gave). The RPM versions (4.1/4.2) mentioned in the URL are quite ancient (circa RHL 8/9). Even on those ancient systems, /var/lib/rpm requires access to root:root (or rpm:rpm) in order to modify an rpmdb

Re: [Rpm-maint] [rpm-software-management/rpm] multiple signatures support? (#189)

There is nothing stopping other applications from using detached signatures on *.rpm files as necessary. RPM cannot carry one (or multiple) signatures within signed plaintext. (aside) In principle a different ping-pong like signing could be attempted to ensure that both signature/metadata

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

OK, here is one way to install a file without leaving the file in the cache. The simplest explanation is to show the --rpmiodebug log (interspersed with some debugging): *** Fopen ufdio path /usr/share/doc/hello-2.0/README;58ea9807 fmode w+.ufdio ***

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Here is an updated patch, including the bare minimum changes with a %_periodic_fsync "opt-out" macro to the "production" rpmfiArchiveReadToFilePsm() to use fdatasync+posix_fadvise+fsync. I've left all the debugging gook in the patch to document the --rpmiodebug trace and to illustrate the

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

As promised, here is a proof-of-concept patch to permit using either fwrite+fsync or mmap+madvise+msync while writing files onto the file system from a *.rpm package. The patch passes "make check" (with the current known failures) on tip. There are various switches to enable/disable the added

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

I knew if I wrote a patch, better answers would instantly appear. There is fadvise(FADV_DONTNEED) wrto fsync(2) analogous to madvise(MADV_DONTNEED) wrto msync(2). I will add to my patch in the next few days for measurement, todo++. The easy answer for your "Chef installs packages and my

Re: [Rpm-maint] [rpm-software-management/rpm] Adding high precision event time stamps to RPM (#197)

Attached is patch to extend RPMTAG_{BUILDTIME,INSTALLTIME,INSTALLTID} to add a 2nd element that contains a struct timespec tspec.tv_nsec 2nd element. There are 2 new getters/setters added to the rpmts API: rpm{Get,Set}Tid2(). About the only subtlety (so far) is the need (for rpmdb "legacy

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Yes, with about 20 other similar articles and implementations, some even accurate. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Wall clock time measurements for an RPM command (this is RPM5, but the I/O trace is identical to rpm.org): `sudo /usr/bin/time /opt/local/bin/rpm -Uvh --nodeps --noscripts --relocate /=/opt/local/tmp /var/tmp/kernel-core-4.11.0-0.rc6.git2.1.fc27.x86_64.rpm` BASE 2.26user 0.43system

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

@pmatila: if you *do* attempt to configure +/- fsync(2) somehow, then durability is likely the better approach. By "durability", I mean "this file is essential to booting" (and hence is worth paying the cost of fsync(2) to ensure that the file is correctly written to disk). Some simple

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Yes, fdatasync+fadvise_syncfs removes pages from system cache: ` --> Fincore(0x61408e40) fdno 16 path /opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f7761a <-- Fincore(0x61408e40) fdno 16 rc 0 <-- fdClose(0x61408e40) rc 0| UFD -1 fp 0x61204e40

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

A measurement of the impact is needed first, enabler/disabler configuration can always be added. In general, RPM does best with "opt-out", not "opt-in" policies. There may be other changes, such as doing the fdatasync() less often, may be useful too. -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

If you are arguing for "opt-in" rather than "opt-out" by default when a macro is not explicitly set, then change the "1" in this line: ` _periodic_fsync = rpmExpandNumeric("%{?_periodic_fsync}%{!?_periodic_fsync:1}"); ` Otherwise set %_periodic_fsync as you wish. There wasn't

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

Another issue that might make debugedit.c more portable is similar to inlining DW_FORM* instead of `#include ` Instead of using EU version dependent #defines for strtab_init() etc (which needs -lebl), it may be more "portable" to just inline the routines and maintain a copy in debugedit.c.

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

Yes all the patches were reversed. Meanwhile I just sorted the first patch with Mark Hatle (aka "fray"). The segfault is not reproducible with EU-0.168. The hex printing was an attempt to avoid pgpHexStr() which differs between rpm.org and rpm5.org. You explained away the valid_file() check

[Rpm-maint] [rpm-software-management/rpm] Retrofitting a format version on *.rpm packaging. (#172)

When originally written, the rpmlead contained information used to identify a *.rpm. Twenty years later, most of the rpmlead information is de facto and ad hoc and historical. Progress using other information, like rpmlib(...) tracking dependencies, has managed to associate new "features"

[Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

rpm5.org carries this patch from Yocto: https://patchwork.openembedded.org/patch/46887 which likely should be added when updating debug edit.c soon. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

For rpm portability with older versions of elfutils, this harmless sanity check might be useful: (excuse the different hash functions in use in the patch snippet below) @@ -1533,40 +1493,12 @@ /* Now format the build ID bits in hex to print out. */ { const uint8_t * id = (uint8_t

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

Portability of debugedit.c on systems that do not support #include (solaris, *BSD) can be achieved by duplicating certain DWARF standard constants: -#ifndef EM_AARCH64 -#define EM_AARCH64 183 /* ARM AARCH64 */ -#endif -#ifndef R_AARCH64_ABS32 -#define R_AARCH64_ABS32 258

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

debugedit.c --help can be improved (and potentially internationalized) *shrug* @@ -1323,16 +1290,15 @@ static struct poptOption optionsTable[] = { { "base-dir", 'b', POPT_ARG_STRING, _dir, 0, -N_("base build DIRECTORY of objects"), N_("DIRECTORY") }, + "base build directory of

[Rpm-maint] [rpm-software-management/rpm] Print exit code with rpm -vv (#195)

This trivial patch helps diagnose rpm behavior while developing or testing. ` diff --git a/rpmqv.c b/rpmqv.c index 8655776..4f448c2 100644 --- a/rpmqv.c +++ b/rpmqv.c @@ -354,5 +354,7 @@ int main(int argc, char *argv[]) rpmcliFini(optCon); +rpmlog(RPMLOG_DEBUG, "exit code: %d\n",

Re: [Rpm-maint] [rpm-software-management/rpm] ELF file conflict 'color' resolution for tri-lib systems (#193)

Resolving file "colors" affects more than MIPS64 n32: the x32 format also uses (1 <<2) color bit and (potentially) needs RPM to choose between executables that are compiled differently. AFAIK, PLD has succeeded in adding support for x32 to RPM5, if you need patches that have been tested in

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

Thanks for measurements! Could you also attempt to measure the effect on RPM too? Wall clock time is sufficient, as the time necessary to perform an upgrade is what most RPM users are sensitive to. Your graphs (presumably on SSD?) show smaller/periodic writes w fsync enabled, and larger/delayed

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

The buffer is 32 * BUFSIZ )= 128Kb) not 32Kb. Sorry for the confusion. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

One last note: doing fdatasync once, before fadvise+fsync, not after every write is ~10x faster: ` FDIO: 30 writes, 3763601 total bytes in 0.003558 secs FDIO: 1 allocs, 3763601 total bytes in 0.24 secs FDIO: 1 dsyncs,0 total bytes in 0.074081 secs FDIO: 1

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

@cgwalters: easy enough to wrap syncfs(2) into a measurement harness: ` ==> Fclose(0x61408e40) | LIBIO 0x61204fc0(-1) fdno -1 | UFD 16 fp 0x61204fc0 --> Fincore(0x61408e40) fdno 16 path /opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66 <--

Re: [Rpm-maint] [rpm-software-management/rpm] RPM+NDB doesn't implement rpmpkgSetFsync() (#288)

Changing nab_SetFsync() to do the obvious (but perhaps naive) passing: ``` static void ndb_SetFSync(rpmdb rdb, int enable) { struct ndbEnv_s *ndbenv = rdb->db_dbenv; if (ndbenv->xdb) rpmxdbSetFsync(ndbenv->xdb, enable); if (ndbenv->pkgdb) rpmpkgSetFsync(ndbenv->pkgdb,

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Offer LMDB as an alternative engine to BDB for rpmdb (#128)

@leo-yuriev: libmdbx looks very nice, but there are several issues that need to be fixed in an rpmdb schema before attempting better back ends. See other issues I have reported against RPM ... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] hdrNum should be stored in big-endian, not native endian, for btree retrieval (#287)

Question #5 in the Berkeley DB Access Method FAQ says this: ``` I'm using integers as keys for a Btree database, and even though the key/data pairs are entered in sorted order, the page-fill factor is low. This is usually the result of using integer keys on little-endian architectures such as

[Rpm-maint] [rpm-software-management/rpm] RPM backend performance is limited by arrays of hdrNum's (#290)

The following callgraphs for BDB/LMDB/NDB all show a common hotspot retrieving arrays of hdrNum's from indices. The performance problem shows up worst on add/del operations, where a RMW loop has to be performed to add/del a hdrNum item to an array. The array is then sorted (and perhaps

[Rpm-maint] [rpm-software-management/rpm] RPM+NDB should implement ndb_dump/ndb_load/ndb_stat for database dumps. (#289)

Converting databases is painful, particularly if both forward/backward conversions are desired. BerkeleyDB has a set of utilities to do this conversion, LMDB uses an almost identical set of utilities. While rpmdb --importdb/--exportdb are a high performing KISS improvement on the BDB-style

Re: [Rpm-maint] [rpm-software-management/rpm] hdrNum should be stored in big-endian, not native endian, for btree retrieval (#287)

NDB appears to not deal with database portability between LE and BE machines. So I'll leave that patch for NDB to implement. The change from native -> big endian is likely a noop anyways because NDB is based on hashed access of hdrNum primary keys. -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Offer LMDB as an alternative engine to BDB for rpmdb (#128)

Google Translate works just fine: time to become pluralistic even if you can't crack a joke in Russian. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

The last patch to RPM_CHECK_LIB is appropriate for "opt-out" builds (i.e. --without-lmdb). Leave "no" if willing to explicitly "opt-in" by passing --with-lmdb to configure. For a serious test in CI, arrange for the macro %_db_backend lmdb to be set by default. Almost certainly you will also

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

The attached patch adds allocate/fadvise/fdatasync/fsync functions in rpmio with the calls on I/O intensive install code paths. In addition, two flags have been added to Fopen(): 'S' to do fsync-on-close, and '?' to do per-fd debugging. Simple benchmarks for a kernel package installation

Re: [Rpm-maint] [rpm-software-management/rpm] Add missing "il" to header sanity check error message (9a6a95b)

How is this failure considered a "tag number mismatch"? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Macro arguments do not support quoting (#222)

Leave bad enough alone imho. Every macro templating language in the world has quoting problems: its the nature of the beast. And a Principle of Least Surprise "Do what bash does!" sets expectations that are wildly out of line with reality. FWIW, the only place that macros MUST be used

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

Note that a package produced by rpm 4.13 reproduced the problem. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

Sigh ... there are much better fixes for ^C handling than reopening an rpmdb (where ^C will *still* be disabled while installing). For starters, the signal handling in RPM was designed to be polled at code points where the rpmdb is known to be consistent (for a graceful exit with an rpmdbClose

Re: [Rpm-maint] [rpm-software-management/rpm] Convert output to the current locale (#250)

The more modern interface is iconv(3). Here is the basic algorithm. The right implementation adds what is known as a header format extension to convert the encoding of any output displayed by rpm --query (aka headerSprintf() ...). hth

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

@mischroe: DB_PRIVATE has its own associated durability/signal problems, including that ^C is masked while installing, and with the transaction lock being for exclusive writers, not readers. Truly ^C handling needs to be solved intelligently, not with convoluted contortions to BDB

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

I'm happy it works fine for SuSE: I'm sure you have tested. Meanwhile ^C exiting and calling rpmdbClose is a problem that needs to be solved without convoluted BDB configuration changes. Its tiresome to continuously revisit this issue, in patches and bug reports. -- You are receiving this

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

The RPM_CHECK_LIB macro: add to acinclude.m4. [rpm_check_lib.m4.gz](https://github.com/rpm-software-management/rpm/files/1156106/rpm_check_lib.m4.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

There are (at least) 3 new FSE compression formats that have been released in the past few years: zstd (Facebook), brotli (Google), lzfse (Apple). The new compressors have better compression ratios and speed than gzip/zlib. And while then compression rations is still not as good as XZ/LZMA, the

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

Here are URL's for zstd, brotli, and lzfse" - [https://github.com/facebook/zstd.git](url) - [https://github.com/google/brotli.git](url) - [https://github.com/lzfse/lzfse.git](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

[Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

Projects that use autoconf are expected to detect build prerequisites and libraries. This usually leads to a set of ad hoc de facto m4 macros that deal with various details like API and path incompatibilities, much of which is platform and os and distro dependent. Various standard tools (like

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

Here is a moderately complex real world example of how to use: detecting ZSTD either externally or internally. In configure.ac: ``` dnl # Facebook Zstd RPM_CHECK_LIB( [Facebook Zstd], [zstd], [zstd], [ZSTD_versionNumber], [zstd.h], [no,external:internal:none], [zstd:lib:lib], [

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

The often stated public policy here with RPM4 is "Compatibility with RPM5 is a negative goal." Don't hold your breath for a patch: I will generate and distribute a patch as soon as I understand the full consequences of recent changes to the RPM format. Yes, Yocto issues are well known. The

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

That is a package produced by RPM5: ``` $ /opt/local/bin/rpm -qp --yaml ./capsule-01.00.12-r0.core2_64.rpm | grep Rpmversion Rpmversion: 5.4.16 ``` The package uses a non-repudiable signature and verifies fine: ``` $ /opt/local/bin/rpm -qvvp ./capsule-01.00.12-r0.core2_64.rpm ... D: PUB:

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

btw, this command is sufficient to identify the rpm version that built the package: `rpm -qp --qf '%{rpm version}\n' intel-aero-repo-1.4-r0.corei7_64.rpm` It looks the arguments passed used in the exact_size check are incorrect (the "ril" and "rdl" values in the error message seem wrong).

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

The RPM+UUID patch in issue #269 illustrates another, more subtle usage feature of RPM_CHECK_LIB. There are two uuid.h files on a linux system: one from ossp-uuid, the other from ext4. Using --with-uuid=system (rather than --with-uuid=external) finds the correct uuid.h. Think about the

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

Specific to OS X portability: there are no efforts to port RPM to OS X (or Mac OS) natively (i.e. using only utilities that come from the OS vendor) that I am aware of for ~15 years. All existing porting efforts involve Fink, MacPorts or Homebrew, and so "portability" includes duplicating

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

Again: my 1st suggestion was for a removal from git to avoid an annoying warning when the file (as always) is replaced by autoreconf. I added historical context about why an otherwise generated file was added to CVS (at the time), and have personally seen the annoyances that happen when

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

FWIW, I have *repeatedly* argued for native (i.e. --prefix=/usr, etc etc) installation on OS X to no avail. I have also successfully ported and run RPM on OS X for many years. Meanwhile I return to my original statement: > ... there are no efforts to port RPM to OS X (or Mac OS) natively ...

Re: [Rpm-maint] [rpm-software-management/rpm] add RPM_CHECK_LIB m4 macro (from jbj) (#282)

Only acinclude.m4 is needed AFAIK. Apologies for the 2 patches at issue #257: I forgot about the handful of other simple macros that RPM_CHECK_LIB needs/uses when I first posted. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

Next plateau: Convert a BDB -> LMDB database. Two simple scripts to assist with the conversion. [bdbexport.sh.gz](https://github.com/rpm-software-management/rpm/files/1180126/bdbexport.sh.gz) [lmdbimport.sh.gz](https://github.com/rpm-software-management/rpm/files/1180129/lmdbimport.sh.gz) Note

[Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

There have been years of vitriol about Berkeley DB and numerous suggested database replacements for RPM. LMDB is perhaps the best choice because LMDB is most similar to Berkeley DB in API and design. Attached is a patch to add an LMDB backend to RPM. I will supply an updated lmdb.c

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

One has to factor in the context of "choice": I'm tired of years of belly aching about RPM databases from the uninformed. That's a very different context than "most similar". The flaws in the rpmdb implementation have nothing whatsoever to do with what database implementation is chosen. --

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

Next plateau: Retrieve header from secondary -> primary lookup: DONE. (stoopid brain fart from rescoping the byte swapped hdrNum with an address in a MDB_val) [lmdb.c-20170726.1.gz](https://github.com/rpm-software-management/rpm/files/1177633/lmdb.c-20170726.1.gz) Now for package management

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

Next plateau: Ensure --query/--install/--rebuilddb are functional: DONE (there are fundamental design flaws (and bugs) with the current RPM+BDB: there is no concept of "durability" as in ACID that can be discussed meaningfully for starters.) On to conversion: there's almost no reason to deal

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

A (lightly tested) RPM+ZSTD patch. You will need RPM_CHECK_LIB() from the acinclude.m4 comment #2 on issue #255. [rpm_zstd.patch.gz](https://github.com/rpm-software-management/rpm/files/1160163/rpm_zstd.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

Hmmm ... its a reasonable guess that failures on BE platforms might be related to endianness, but damfino how endianness becomes an issue on an octet oriented I/O API thru python bindings. I's suggest looking closely at the return codes from *seek/*tell which have never been well supported in

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

Either set _romiodebug = -1, or (even better) fish out the change to set the FD_t specific debugging flag in cvtfmode() if Fopen(path, "r?.fpio") is the invocation. There's really no other way to set FD_t specific debugging because FD_t is totally opaque in rpmio/rpmio.c -- You are receiving

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

Hmmm ... endianness *does* become an issue when a int64 is somehow returned through an int32: LE will Just Work (for small offers) and BE will fail through truncation. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

You might just as easily drop mkinstalldirs. The original motivation for including mkinstalldirs was from last century: certain ancient SysV non-linux platforms lacked "mkdir -p". Carrying mkinstalldirs was the solution. -- You are receiving this because you are subscribed to this thread.

[Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

UUID's provide a common format for identification and database retrieval. The attached patch adds a header tag format to RPM queries. - UUIDv1 time stamps (for events like build/install times): ``` $ ./rpm -q --qf '%{buildtime:uuidv1}\n' bash c60fd500-bc0a-11e7-804e-003048b801de ``` - UUIDv3

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

[rpm_uuid.patch.gz](https://github.com/rpm-software-management/rpm/files/1163437/rpm_uuid.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

There are 3 issues: 1) mkdir -p isn't portable and so mkinstalldirs is needed 2) mkinstalldirs is distributed in /usr/lib/rpm 3) mkinstalldirs is copied and tracked in git or other VCS My comment was wrto issue #3: mkinstalldirs was originally added to RPM in order to assuage the bosses

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

Other nitpicks in the geeky error message: the check has nothing to do with a "tag number mismatch", and lacks an "il" identifier that might have helped diagnose the problem slightly more rapidly. OTOH, I'm not sure anyone can diagnose a problem in the binary header format without a

Re: [Rpm-maint] [rpm-software-management/rpm] Check for 'gettext' during configure script (#245)

Did you build with --disable-nls? There is no way for RPM configure to guess whether uninstalled get text is bug of a feature: YMMV, everyone's does. ``` ./configure --help | grep nls --disable-nls do not use Native Language Support ``` -- You are receiving this because you are

[Rpm-maint] [rpm-software-management/rpm] Adding namespaces for macros (#246)

Macro templating/configuration in RPM has become MUST'VE for building. Meanwhile, there are no conventions for sorting out the definitions, and the issues related to loading all macros all the time are getting unmanageable. E.g. here are the macros defined on a relatively recent Fedora box:

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Pre-validate (all) header data for semantic consistency (#242)

As long as --nosignature and --nodigest are commonly used everywhere -- including in depsolvers -- you can setup an internal table to associate the data type with the tag and never access the field in the header. -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

The attached patch does a rude-and-crude libeio pipe driven event loop to ramie. The patch adds an 'N' (mnemonic: O_NONBLOCK) flag to Fopen() flags. The libeio loop is lazily started, and atexit(3) is used to wait for all events to be completed. The operations of fdatasync/fadvise/fsync are

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

There is one other alternative: libcoro (also by Marc Lehman). If the only problem I was trying to solve was making fdatasync+fsync asynchronous, then libcoro would be the best choice. My patch essentially reduces libeio to co-routine functionality. libcoro does not require a scheduler, nor

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

Yes, there is libevent, then libev+libeio, and now libuv, all tuned towards slightly different goals. libeio is narrowly targeted at asynchronous system calls, can be embedded, and is generally lean-and-mean and well tested. But go shopping for alternatives if you wish: I chose libeio after

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

The main reason why libevent was not used: * Marc Lehmann chose to move on from libevent -> libev+libeio and strip out complex buffer handling. Small and targeted and embeddable are all KISS; automating buffer allocation handling is hardly useful or necessary to rpmio. The

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

The only sane way to add an event loop within RPM is through embedding, at which point maintenance is with RPM, not through actively maintained external libraries. libeio is the only reasonable embeddable choice. Extending external libeio with additional system calls is trickier with external

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

Some minor tinkering on the fdatasync -> fsync pipeline with code like this: ``` pthread_yield(); if (eio_npending()) rc = eio_poll(); ``` to minimize the number of calls to eio_poll() (which always processes at least one request) gives these timings ``` $ sudo /usr/bin/time

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

And one more try at speeding up fsync-on-close by adding more threads that can block: ``` eio_set_min_parallel(512); eio_set_max_idle(512); ``` With 512 (the default is 4) backend threads that can block, the kernel install time is ``` $ sudo /usr/bin/time ./rpm -U

  1   2   3   4   5   6   7   >