Re: [Rpm-maint] relicensing for Maximum RPM

2009-09-07 Thread Jeff Johnson
of the ones who worked for rht at the time they were writing it. So I suspect Paul and jbj's contributions are (c) Red Hat inc. But Matthias and Ville are (c) themselves, unless otherwise disclaimed. You may suspect all you wish, but you better check dates before making false claims in

[Rpm-maint] Keep my name out of your discussions

2010-06-21 Thread Jeff Johnson
Re: BTW, Jeff J. must have changed his mind at some point, because rpm5 allows short-circuiting of -bb. You have no right to discuss the state of my mind. In fact, you haven't a clue. Keep my name out of your gossipy garbage. 73 de Jeff ___

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

2017-03-28 Thread Jeff Johnson
My bad for not noticing. Good that padding has been added by adding (apparently) 0x1000 to the data size: `0060: 8ead e801 0006 1074 ...t` Adding a "free space" tag (as @rpm5.org) provides an indication that the signature header is padded. Meanwhile, the

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

2017-03-28 Thread Jeff Johnson
Ah, you have added RPMSIGTAG_RESERVEDSPACE already which I was not expecting. The (minor but annoying problems) with the assignment to 1008 is that RPMSIGTAG_RESERVEDSPACE collides with RPMTAG_INSTALLTIME, and will not sort correctly to the end of an immutable region data section as a "free

[Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

2017-03-28 Thread Jeff Johnson
The recently added RPMTAG_PAYLOADDIGEST computes a digest on the compressed payload. Perhaps now is the time to change the plaintext used to be the uncompressed rather than the compressed archive payload? The benefit of the digest on the uncompressed archive is that the digest becomes immune

[Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

2017-03-27 Thread Jeff Johnson
The signature header contains (optional) information to verify package integrity. So new tags are added when packages are signed, and a new signature header is created and inserted into a *.rpm package file. However, the insertion forces a rewrite of the metadata header and payload, which for

[Rpm-maint] [rpm-software-management/rpm] Adding/moving RPMTAG_PUBKEYS to the signature header (#182)

2017-03-27 Thread Jeff Johnson
RPM package files can include public keys in the metadata header. So in principle, a package can include the public key used to sign a package and verify the package signature when reading package headers. The RPMTAG_PUBKEYS array is also within the signed immutable region and cannot be

[Rpm-maint] [rpm-software-management/rpm] Tags in the signature header are not sorted (#181)

2017-03-25 Thread Jeff Johnson
This is the hex dump of the createrepo_c Archer-3.4.5-6.x86_64.rpm signature header. RPMSIGTAG_SHA1 follows RPMSIGTAG_MD5 at offset 0xa0, forcing a sort while loading the signature header. _ 0060: 8ead e801 0005 0054 ...T 0070: 003e 0007

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-04 Thread Jeff Johnson
Do you have measurements of the effect of fsync()? The last time sync/fdatasync were suggested for RPM that I am aware of was ~2011 in Mandriva. The issue comes up every few years this century ... Here was the effect of adding fsync/fdatasync to RPM, measure on otherwise identical system with

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-04 Thread Jeff Johnson
Good if doing in fsm.h ( I haven't looked). Meanwhile a 50x slower installation is intolerable for a package manager even if opt-in. Nor can an application like RPM be expected to detect whether the underlying filesystem is on SSD or USB or rotational disk and adapt accordingly: there simply

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
FYI: several notes (after looking at the patch) 1) The measurements I cited above are using mmap(2) to decompress directly into pages (avoiding a buffer copy) marked with MADV_DONTNEED (which permits delayed/larger writes) to avoid a buffer copy. In addition, there is a check on file size of

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
For completeness (and humor ;-), there is also O_DIRECT. See [https://lkml.org/lkml/2007/1/10/233](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
There is a modestly obscure means to attach *_DONTNEED (to achieve kernel delayed writes and removing pages when I/O is complete) using the existing rpm.org rpmfiArchiveReadToFilePsm() Fwrite->write(3) loop calling setvbuf(3) immediately after Fopen on the fopencookie(3) FILE *. (aside)

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
There is an alternative (but more complicated) lazy implementation that might be attempted by doing a fsync in Fwrite (or fdWrite) when the accumulated I/O passes a certain mark. H ... A general solution to add a resource cap on all rpmio I/O could also be attempted (but isn't easy).

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
Hmmm ... glibc (which uses mmap(2) for stdio if requested) would benefit from being able to mark the stdio buffer-in-use with *_DONTNEED using madvise/posix_fadvise. At a minimum, cvtfmode() in rpmio/rpmio.c would need to be extended to pass the glibc-peculier 'm' flag through to fopen(3)

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
Another issue is that certain file systems (NFS? DFS? I forget ...) also delay I/O operations until close(2). What I recall is that an application MUST check the return code of close(2) in order to ensure that the I/O was actually performed: it isn't sufficient to check the return code of

Re: [Rpm-maint] [rpm-software-management/rpm] Adding high precision event time stamps to RPM (#197)

2017-04-11 Thread Jeff Johnson
Um, this patch is not ready to be pulled and blindly applied (but is a complete proof-of-concept implementation afaik). A similar has been used @rpm5.org for ~8 years. There has been no discussion or detectable interest, including: * should usecs/nsecs be used? is there a need to accommodate

Re: [Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

2017-04-02 Thread Jeff Johnson
(answering my own question) Short answer: digesting compressed payload instead of compressing the digested payload is expedient. Long answer: RPM goes to some lengths to preserve streaming-without-peeking-at-magic for *.rpm packages. The usual RPM approach is/was to add/save tags in the

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-11 Thread Jeff Johnson
Thanks for measurements. So fsync on rotating DASD is ~9x slower for your db-gcc-runtime test (IIUC). I'm now investigating the effect of fallocate(2) to reserve disk blocks (and prevent ENOSPC returns) on the code path we are iterating upon. News at 11pm ;-) -- You are receiving this

[Rpm-maint] [rpm-software-management/rpm] Colorized rpmlog() output (+PATCH) (#198)

2017-04-11 Thread Jeff Johnson
SSIA Have fun! [rpm+color.patch.gz](https://github.com/rpm-software-management/rpm/files/915427/rpm.color.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Make 'rpm -V' more resistent against rpmdb manipulations (#196)

2017-04-11 Thread Jeff Johnson
FWIW, rpm.org has the first (but insufficient) step to hardening forensics on a compromised system. E.g Here is rpm-4.13.90 output of "rpm -Vvv popt" ` ... D: read h# 27079 Header V3 RSA/SHA256 Signature, key ID 81b46521: OK D: Plugin: calling hook init in syslog plugin D: Plugin: calling

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: option for initial-only content in files (#152)

2017-04-12 Thread Jeff Johnson
Add * update by patching and make the policies platform and machine specific. Then call the implementation something catchier than ansible or DevOps. Seriously: configuration != package management. -- You are receiving this because you commented. Reply to this email directly or view it on

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-06 Thread Jeff Johnson
@jayzmh: major invasive change to ... RPM? or something else? I'll happily write the code for rpm.org RPM if you wish. I fail to see how reverting to what RPM used to do (i.e. use mmap with MADV_DONTNEED and most importantly for you using msync(MS_ASYNC) which schedules flush to disk like

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-06 Thread Jeff Johnson
@jayzmh - add a static variable to expand the macro as a "one shot" ... there are many places in rpm that do that, hurts little (until rpm can capture global configuration object). Meanwhile I recommend the *_DONTNEED mmap implementation instead of fsync. RPM used that for years, and

Re: [Rpm-maint] [rpm-software-management/rpm] Make 'rpm -V' more resistent against rpmdb manipulations (#196)

2017-04-09 Thread Jeff Johnson
Hmmm ... its not clear what exploit is used (from just reading the file at the URL you gave). The RPM versions (4.1/4.2) mentioned in the URL are quite ancient (circa RHL 8/9). Even on those ancient systems, /var/lib/rpm requires access to root:root (or rpm:rpm) in order to modify an rpmdb

[Rpm-maint] [rpm-software-management/rpm] Adding high precision event time stamps to RPM (#197)

2017-04-09 Thread Jeff Johnson
RPM uses unix seconds-since-the-epoch for various event timestamps (e.g. RPMTAG_BUILDTIME/RPMTAG_INSTALLTIME) stored as a RPM_INT32_TYPE scalar in a Header. Because scalar integers in RPM are (essentially) stored as an array with a single element, the tags used to store event time stamps can

Re: [Rpm-maint] [rpm-software-management/rpm] multiple signatures support? (#189)

2017-04-09 Thread Jeff Johnson
There is nothing stopping other applications from using detached signatures on *.rpm files as necessary. RPM cannot carry one (or multiple) signatures within signed plaintext. (aside) In principle a different ping-pong like signing could be attempted to ensure that both signature/metadata

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-09 Thread Jeff Johnson
OK, here is one way to install a file without leaving the file in the cache. The simplest explanation is to show the --rpmiodebug log (interspersed with some debugging): *** Fopen ufdio path /usr/share/doc/hello-2.0/README;58ea9807 fmode w+.ufdio ***

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-09 Thread Jeff Johnson
Here is an updated patch, including the bare minimum changes with a %_periodic_fsync "opt-out" macro to the "production" rpmfiArchiveReadToFilePsm() to use fdatasync+posix_fadvise+fsync. I've left all the debugging gook in the patch to document the --rpmiodebug trace and to illustrate the

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-08 Thread Jeff Johnson
As promised, here is a proof-of-concept patch to permit using either fwrite+fsync or mmap+madvise+msync while writing files onto the file system from a *.rpm package. The patch passes "make check" (with the current known failures) on tip. There are various switches to enable/disable the added

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-08 Thread Jeff Johnson
I knew if I wrote a patch, better answers would instantly appear. There is fadvise(FADV_DONTNEED) wrto fsync(2) analogous to madvise(MADV_DONTNEED) wrto msync(2). I will add to my patch in the next few days for measurement, todo++. The easy answer for your "Chef installs packages and my

Re: [Rpm-maint] [rpm-software-management/rpm] Adding high precision event time stamps to RPM (#197)

2017-04-10 Thread Jeff Johnson
Attached is patch to extend RPMTAG_{BUILDTIME,INSTALLTIME,INSTALLTID} to add a 2nd element that contains a struct timespec tspec.tv_nsec 2nd element. There are 2 new getters/setters added to the rpmts API: rpm{Get,Set}Tid2(). About the only subtlety (so far) is the need (for rpmdb "legacy

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
Yes, with about 20 other similar articles and implementations, some even accurate. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-16 Thread Jeff Johnson
Wall clock time measurements for an RPM command (this is RPM5, but the I/O trace is identical to rpm.org): `sudo /usr/bin/time /opt/local/bin/rpm -Uvh --nodeps --noscripts --relocate /=/opt/local/tmp /var/tmp/kernel-core-4.11.0-0.rc6.git2.1.fc27.x86_64.rpm` BASE 2.26user 0.43system

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-16 Thread Jeff Johnson
@pmatila: if you *do* attempt to configure +/- fsync(2) somehow, then durability is likely the better approach. By "durability", I mean "this file is essential to booting" (and hence is worth paying the cost of fsync(2) to ensure that the file is correctly written to disk). Some simple

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-19 Thread Jeff Johnson
Yes, fdatasync+fadvise_syncfs removes pages from system cache: ` --> Fincore(0x61408e40) fdno 16 path /opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f7761a <-- Fincore(0x61408e40) fdno 16 rc 0 <-- fdClose(0x61408e40) rc 0| UFD -1 fp 0x61204e40

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-13 Thread Jeff Johnson
A measurement of the impact is needed first, enabler/disabler configuration can always be added. In general, RPM does best with "opt-out", not "opt-in" policies. There may be other changes, such as doing the fdatasync() less often, may be useful too. -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
If you are arguing for "opt-in" rather than "opt-out" by default when a macro is not explicitly set, then change the "1" in this line: ` _periodic_fsync = rpmExpandNumeric("%{?_periodic_fsync}%{!?_periodic_fsync:1}"); ` Otherwise set %_periodic_fsync as you wish. There wasn't

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

2017-03-06 Thread Jeff Johnson
Another issue that might make debugedit.c more portable is similar to inlining DW_FORM* instead of `#include ` Instead of using EU version dependent #defines for strtab_init() etc (which needs -lebl), it may be more "portable" to just inline the routines and maintain a copy in debugedit.c.

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

2017-03-06 Thread Jeff Johnson
Yes all the patches were reversed. Meanwhile I just sorted the first patch with Mark Hatle (aka "fray"). The segfault is not reproducible with EU-0.168. The hex printing was an attempt to avoid pgpHexStr() which differs between rpm.org and rpm5.org. You explained away the valid_file() check

[Rpm-maint] [rpm-software-management/rpm] Retrofitting a format version on *.rpm packaging. (#172)

2017-03-07 Thread Jeff Johnson
When originally written, the rpmlead contained information used to identify a *.rpm. Twenty years later, most of the rpmlead information is de facto and ad hoc and historical. Progress using other information, like rpmlib(...) tracking dependencies, has managed to associate new "features"

[Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
rpm5.org carries this patch from Yocto: https://patchwork.openembedded.org/patch/46887 which likely should be added when updating debug edit.c soon. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
For rpm portability with older versions of elfutils, this harmless sanity check might be useful: (excuse the different hash functions in use in the patch snippet below) @@ -1533,40 +1493,12 @@ /* Now format the build ID bits in hex to print out. */ { const uint8_t * id = (uint8_t

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
Portability of debugedit.c on systems that do not support #include (solaris, *BSD) can be achieved by duplicating certain DWARF standard constants: -#ifndef EM_AARCH64 -#define EM_AARCH64 183 /* ARM AARCH64 */ -#endif -#ifndef R_AARCH64_ABS32 -#define R_AARCH64_ABS32 258

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
debugedit.c --help can be improved (and potentially internationalized) *shrug* @@ -1323,16 +1290,15 @@ static struct poptOption optionsTable[] = { { "base-dir", 'b', POPT_ARG_STRING, _dir, 0, -N_("base build DIRECTORY of objects"), N_("DIRECTORY") }, + "base build directory of

[Rpm-maint] [rpm-software-management/rpm] Print exit code with rpm -vv (#195)

2017-04-08 Thread Jeff Johnson
This trivial patch helps diagnose rpm behavior while developing or testing. ` diff --git a/rpmqv.c b/rpmqv.c index 8655776..4f448c2 100644 --- a/rpmqv.c +++ b/rpmqv.c @@ -354,5 +354,7 @@ int main(int argc, char *argv[]) rpmcliFini(optCon); +rpmlog(RPMLOG_DEBUG, "exit code: %d\n",

Re: [Rpm-maint] [rpm-software-management/rpm] ELF file conflict 'color' resolution for tri-lib systems (#193)

2017-04-08 Thread Jeff Johnson
Resolving file "colors" affects more than MIPS64 n32: the x32 format also uses (1 <<2) color bit and (potentially) needs RPM to choose between executables that are compiled differently. AFAIK, PLD has succeeded in adding support for x32 to RPM5, if you need patches that have been tested in

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-08 Thread Jeff Johnson
Thanks for measurements! Could you also attempt to measure the effect on RPM too? Wall clock time is sufficient, as the time necessary to perform an upgrade is what most RPM users are sensitive to. Your graphs (presumably on SSD?) show smaller/periodic writes w fsync enabled, and larger/delayed

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
The buffer is 32 * BUFSIZ )= 128Kb) not 32Kb. Sorry for the confusion. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
One last note: doing fdatasync once, before fadvise+fsync, not after every write is ~10x faster: ` FDIO: 30 writes, 3763601 total bytes in 0.003558 secs FDIO: 1 allocs, 3763601 total bytes in 0.24 secs FDIO: 1 dsyncs,0 total bytes in 0.074081 secs FDIO: 1

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
Here are some measurements (on RPM5, but its essentially the same implementation). The system is Fedora 25, the disk is 7200rpm rotating media, dual 4 core xeons, etc. The System.map file in the current RawHide kernel-core file was used to measure the time taken by the various system calls

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-18 Thread Jeff Johnson
@cgwalters: easy enough to wrap syncfs(2) into a measurement harness: ` ==> Fclose(0x61408e40) | LIBIO 0x61204fc0(-1) fdno -1 | UFD 16 fp 0x61204fc0 --> Fincore(0x61408e40) fdno 16 path /opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66 <--

Re: [Rpm-maint] [rpm-software-management/rpm] RPM+NDB should implement ndb_dump/ndb_load/ndb_stat for database dumps. (#289)

2017-07-31 Thread Jeff Johnson
Bundling into rpmdb would be fine: just there needs to be a common interchange format. E.g. libvirt uses db_dump and is opposed to NDB because it lacks an equivalent. The textual non-hex format is occasionally useful as well (but with digests/signatures one cannot just edit the database). --

Re: [Rpm-maint] [rpm-software-management/rpm] RPM+NDB doesn't implement rpmpkgSetFsync() (#288)

2017-07-29 Thread Jeff Johnson
Changing nab_SetFsync() to do the obvious (but perhaps naive) passing: ``` static void ndb_SetFSync(rpmdb rdb, int enable) { struct ndbEnv_s *ndbenv = rdb->db_dbenv; if (ndbenv->xdb) rpmxdbSetFsync(ndbenv->xdb, enable); if (ndbenv->pkgdb) rpmpkgSetFsync(ndbenv->pkgdb,

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Offer LMDB as an alternative engine to BDB for rpmdb (#128)

2017-07-29 Thread Jeff Johnson
@leo-yuriev: libmdbx looks very nice, but there are several issues that need to be fixed in an rpmdb schema before attempting better back ends. See other issues I have reported against RPM ... -- You are receiving this because you commented. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] hdrNum should be stored in big-endian, not native endian, for btree retrieval (#287)

2017-07-29 Thread Jeff Johnson
Question #5 in the Berkeley DB Access Method FAQ says this: ``` I'm using integers as keys for a Btree database, and even though the key/data pairs are entered in sorted order, the page-fill factor is low. This is usually the result of using integer keys on little-endian architectures such as

[Rpm-maint] [rpm-software-management/rpm] RPM backend performance is limited by arrays of hdrNum's (#290)

2017-07-29 Thread Jeff Johnson
The following callgraphs for BDB/LMDB/NDB all show a common hotspot retrieving arrays of hdrNum's from indices. The performance problem shows up worst on add/del operations, where a RMW loop has to be performed to add/del a hdrNum item to an array. The array is then sorted (and perhaps

[Rpm-maint] [rpm-software-management/rpm] RPM+NDB should implement ndb_dump/ndb_load/ndb_stat for database dumps. (#289)

2017-07-29 Thread Jeff Johnson
Converting databases is painful, particularly if both forward/backward conversions are desired. BerkeleyDB has a set of utilities to do this conversion, LMDB uses an almost identical set of utilities. While rpmdb --importdb/--exportdb are a high performing KISS improvement on the BDB-style

Re: [Rpm-maint] [rpm-software-management/rpm] hdrNum should be stored in big-endian, not native endian, for btree retrieval (#287)

2017-07-29 Thread Jeff Johnson
NDB appears to not deal with database portability between LE and BE machines. So I'll leave that patch for NDB to implement. The change from native -> big endian is likely a noop anyways because NDB is based on hashed access of hdrNum primary keys. -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Offer LMDB as an alternative engine to BDB for rpmdb (#128)

2017-07-29 Thread Jeff Johnson
Google Translate works just fine: time to become pluralistic even if you can't crack a joke in Russian. -- You are receiving this because you commented. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-07-29 Thread Jeff Johnson
The last patch to RPM_CHECK_LIB is appropriate for "opt-out" builds (i.e. --without-lmdb). Leave "no" if willing to explicitly "opt-in" by passing --with-lmdb to configure. For a serious test in CI, arrange for the macro %_db_backend lmdb to be set by default. Almost certainly you will also

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-03 Thread Jeff Johnson
The attached patch adds allocate/fadvise/fdatasync/fsync functions in rpmio with the calls on I/O intensive install code paths. In addition, two flags have been added to Fopen(): 'S' to do fsync-on-close, and '?' to do per-fd debugging. Simple benchmarks for a kernel package installation

Re: [Rpm-maint] [rpm-software-management/rpm] Add missing "il" to header sanity check error message (9a6a95b)

2017-08-15 Thread Jeff Johnson
How is this failure considered a "tag number mismatch"? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Macro arguments do not support quoting (#222)

2017-08-14 Thread Jeff Johnson
Leave bad enough alone imho. Every macro templating language in the world has quoting problems: its the nature of the beast. And a Principle of Least Surprise "Do what bash does!" sets expectations that are wildly out of line with reality. FWIW, the only place that macros MUST be used

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-08-15 Thread Jeff Johnson
Note that a package produced by rpm 4.13 reproduced the problem. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

2017-07-13 Thread Jeff Johnson
Sigh ... there are much better fixes for ^C handling than reopening an rpmdb (where ^C will *still* be disabled while installing). For starters, the signal handling in RPM was designed to be polled at code points where the rpmdb is known to be consistent (for a graceful exit with an rpmdbClose

Re: [Rpm-maint] [rpm-software-management/rpm] Convert output to the current locale (#250)

2017-07-13 Thread Jeff Johnson
The more modern interface is iconv(3). Here is the basic algorithm. The right implementation adds what is known as a header format extension to convert the encoding of any output displayed by rpm --query (aka headerSprintf() ...). hth

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

2017-07-14 Thread Jeff Johnson
@mischroe: DB_PRIVATE has its own associated durability/signal problems, including that ^C is masked while installing, and with the transaction lock being for exclusive writers, not readers. Truly ^C handling needs to be solved intelligently, not with convoluted contortions to BDB

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

2017-07-14 Thread Jeff Johnson
I'm happy it works fine for SuSE: I'm sure you have tested. Meanwhile ^C exiting and calling rpmdbClose is a problem that needs to be solved without convoluted BDB configuration changes. Its tiresome to continuously revisit this issue, in patches and bug reports. -- You are receiving this

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-18 Thread Jeff Johnson
The RPM_CHECK_LIB macro: add to acinclude.m4. [rpm_check_lib.m4.gz](https://github.com/rpm-software-management/rpm/files/1156106/rpm_check_lib.m4.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-07-18 Thread Jeff Johnson
There are (at least) 3 new FSE compression formats that have been released in the past few years: zstd (Facebook), brotli (Google), lzfse (Apple). The new compressors have better compression ratios and speed than gzip/zlib. And while then compression rations is still not as good as XZ/LZMA, the

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-07-18 Thread Jeff Johnson
Here are URL's for zstd, brotli, and lzfse" - [https://github.com/facebook/zstd.git](url) - [https://github.com/google/brotli.git](url) - [https://github.com/lzfse/lzfse.git](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

[Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-18 Thread Jeff Johnson
Projects that use autoconf are expected to detect build prerequisites and libraries. This usually leads to a set of ad hoc de facto m4 macros that deal with various details like API and path incompatibilities, much of which is platform and os and distro dependent. Various standard tools (like

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-18 Thread Jeff Johnson
Here is a moderately complex real world example of how to use: detecting ZSTD either externally or internally. In configure.ac: ``` dnl # Facebook Zstd RPM_CHECK_LIB( [Facebook Zstd], [zstd], [zstd], [ZSTD_versionNumber], [zstd.h], [no,external:internal:none], [zstd:lib:lib], [

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-20 Thread Jeff Johnson
Thanks for identifying the specific checkin. RPM4 has changed the definition of what is called an "immutable region" in a signature header. An "immutable region" is identified with an index/data length as well as a trailer which duplicates the same information. The test for an exact_match in

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-20 Thread Jeff Johnson
The often stated public policy here with RPM4 is "Compatibility with RPM5 is a negative goal." Don't hold your breath for a patch: I will generate and distribute a patch as soon as I understand the full consequences of recent changes to the RPM format. Yes, Yocto issues are well known. The

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-20 Thread Jeff Johnson
That is a package produced by RPM5: ``` $ /opt/local/bin/rpm -qp --yaml ./capsule-01.00.12-r0.core2_64.rpm | grep Rpmversion Rpmversion: 5.4.16 ``` The package uses a non-repudiable signature and verifies fine: ``` $ /opt/local/bin/rpm -qvvp ./capsule-01.00.12-r0.core2_64.rpm ... D: PUB:

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-22 Thread Jeff Johnson
btw, this command is sufficient to identify the rpm version that built the package: `rpm -qp --qf '%{rpm version}\n' intel-aero-repo-1.4-r0.corei7_64.rpm` It looks the arguments passed used in the exact_size check are incorrect (the "ril" and "rdl" values in the error message seem wrong).

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-22 Thread Jeff Johnson
The RPM+UUID patch in issue #269 illustrates another, more subtle usage feature of RPM_CHECK_LIB. There are two uuid.h files on a linux system: one from ossp-uuid, the other from ext4. Using --with-uuid=system (rather than --with-uuid=external) finds the correct uuid.h. Think about the

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
Specific to OS X portability: there are no efforts to port RPM to OS X (or Mac OS) natively (i.e. using only utilities that come from the OS vendor) that I am aware of for ~15 years. All existing porting efforts involve Fink, MacPorts or Homebrew, and so "portability" includes duplicating

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
Again: my 1st suggestion was for a removal from git to avoid an annoying warning when the file (as always) is replaced by autoreconf. I added historical context about why an otherwise generated file was added to CVS (at the time), and have personally seen the annoyances that happen when

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
FWIW, I have *repeatedly* argued for native (i.e. --prefix=/usr, etc etc) installation on OS X to no avail. I have also successfully ported and run RPM on OS X for many years. Meanwhile I return to my original statement: > ... there are no efforts to port RPM to OS X (or Mac OS) natively ...

Re: [Rpm-maint] [rpm-software-management/rpm] add RPM_CHECK_LIB m4 macro (from jbj) (#282)

2017-07-27 Thread Jeff Johnson
Only acinclude.m4 is needed AFAIK. Apologies for the 2 patches at issue #257: I forgot about the handful of other simple macros that RPM_CHECK_LIB needs/uses when I first posted. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-27 Thread Jeff Johnson
Next plateau: Convert a BDB -> LMDB database. Two simple scripts to assist with the conversion. [bdbexport.sh.gz](https://github.com/rpm-software-management/rpm/files/1180126/bdbexport.sh.gz) [lmdbimport.sh.gz](https://github.com/rpm-software-management/rpm/files/1180129/lmdbimport.sh.gz) Note

[Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
There have been years of vitriol about Berkeley DB and numerous suggested database replacements for RPM. LMDB is perhaps the best choice because LMDB is most similar to Berkeley DB in API and design. Attached is a patch to add an LMDB backend to RPM. I will supply an updated lmdb.c

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
One has to factor in the context of "choice": I'm tired of years of belly aching about RPM databases from the uninformed. That's a very different context than "most similar". The flaws in the rpmdb implementation have nothing whatsoever to do with what database implementation is chosen. --

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
Next plateau: Retrieve header from secondary -> primary lookup: DONE. (stoopid brain fart from rescoping the byte swapped hdrNum with an address in a MDB_val) [lmdb.c-20170726.1.gz](https://github.com/rpm-software-management/rpm/files/1177633/lmdb.c-20170726.1.gz) Now for package management

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
Next plateau: Ensure --query/--install/--rebuilddb are functional: DONE (there are fundamental design flaws (and bugs) with the current RPM+BDB: there is no concept of "durability" as in ACID that can be discussed meaningfully for starters.) On to conversion: there's almost no reason to deal

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-07-19 Thread Jeff Johnson
A (lightly tested) RPM+ZSTD patch. You will need RPM_CHECK_LIB() from the acinclude.m4 comment #2 on issue #255. [rpm_zstd.patch.gz](https://github.com/rpm-software-management/rpm/files/1160163/rpm_zstd.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

2017-07-19 Thread Jeff Johnson
Hmmm ... its a reasonable guess that failures on BE platforms might be related to endianness, but damfino how endianness becomes an issue on an octet oriented I/O API thru python bindings. I's suggest looking closely at the return codes from *seek/*tell which have never been well supported in

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

2017-07-19 Thread Jeff Johnson
Either set _romiodebug = -1, or (even better) fish out the change to set the FD_t specific debugging flag in cvtfmode() if Fopen(path, "r?.fpio") is the invocation. There's really no other way to set FD_t specific debugging because FD_t is totally opaque in rpmio/rpmio.c -- You are receiving

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

2017-07-19 Thread Jeff Johnson
Hmmm ... endianness *does* become an issue when a int64 is somehow returned through an int32: LE will Just Work (for small offers) and BE will fail through truncation. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
You might just as easily drop mkinstalldirs. The original motivation for including mkinstalldirs was from last century: certain ancient SysV non-linux platforms lacked "mkdir -p". Carrying mkinstalldirs was the solution. -- You are receiving this because you are subscribed to this thread.

[Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-07-20 Thread Jeff Johnson
UUID's provide a common format for identification and database retrieval. The attached patch adds a header tag format to RPM queries. - UUIDv1 time stamps (for events like build/install times): ``` $ ./rpm -q --qf '%{buildtime:uuidv1}\n' bash c60fd500-bc0a-11e7-804e-003048b801de ``` - UUIDv3

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-07-20 Thread Jeff Johnson
[rpm_uuid.patch.gz](https://github.com/rpm-software-management/rpm/files/1163437/rpm_uuid.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
There are 3 issues: 1) mkdir -p isn't portable and so mkinstalldirs is needed 2) mkinstalldirs is distributed in /usr/lib/rpm 3) mkinstalldirs is copied and tracked in git or other VCS My comment was wrto issue #3: mkinstalldirs was originally added to RPM in order to assuage the bosses

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-22 Thread Jeff Johnson
Other nitpicks in the geeky error message: the check has nothing to do with a "tag number mismatch", and lacks an "il" identifier that might have helped diagnose the problem slightly more rapidly. OTOH, I'm not sure anyone can diagnose a problem in the binary header format without a

Re: [Rpm-maint] [rpm-software-management/rpm] Check for 'gettext' during configure script (#245)

2017-06-30 Thread Jeff Johnson
Did you build with --disable-nls? There is no way for RPM configure to guess whether uninstalled get text is bug of a feature: YMMV, everyone's does. ``` ./configure --help | grep nls --disable-nls do not use Native Language Support ``` -- You are receiving this because you are

[Rpm-maint] [rpm-software-management/rpm] Adding namespaces for macros (#246)

2017-07-01 Thread Jeff Johnson
Macro templating/configuration in RPM has become MUST'VE for building. Meanwhile, there are no conventions for sorting out the definitions, and the issues related to loading all macros all the time are getting unmanageable. E.g. here are the macros defined on a relatively recent Fedora box:

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Pre-validate (all) header data for semantic consistency (#242)

2017-06-28 Thread Jeff Johnson
As long as --nosignature and --nodigest are commonly used everywhere -- including in depsolvers -- you can setup an internal table to associate the data type with the tag and never access the field in the header. -- You are receiving this because you are subscribed to this thread. Reply to

  1   2   3   4   5   6   7   8   >