Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
Next plateau: Retrieve header from secondary -> primary lookup: DONE. (stoopid brain fart from rescoping the byte swapped hdrNum with an address in a MDB_val) [lmdb.c-20170726.1.gz](https://github.com/rpm-software-management/rpm/files/1177633/lmdb.c-20170726.1.gz) Now for package management

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
One has to factor in the context of "choice": I'm tired of years of belly aching about RPM databases from the uninformed. That's a very different context than "most similar". The flaws in the rpmdb implementation have nothing whatsoever to do with what database implementation is chosen. --

[Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-26 Thread Jeff Johnson
There have been years of vitriol about Berkeley DB and numerous suggested database replacements for RPM. LMDB is perhaps the best choice because LMDB is most similar to Berkeley DB in API and design. Attached is a patch to add an LMDB backend to RPM. I will supply an updated lmdb.c

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
FWIW, I have *repeatedly* argued for native (i.e. --prefix=/usr, etc etc) installation on OS X to no avail. I have also successfully ported and run RPM on OS X for many years. Meanwhile I return to my original statement: > ... there are no efforts to port RPM to OS X (or Mac OS) natively ...

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
Specific to OS X portability: there are no efforts to port RPM to OS X (or Mac OS) natively (i.e. using only utilities that come from the OS vendor) that I am aware of for ~15 years. All existing porting efforts involve Fink, MacPorts or Homebrew, and so "portability" includes duplicating

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
Again: my 1st suggestion was for a removal from git to avoid an annoying warning when the file (as always) is replaced by autoreconf. I added historical context about why an otherwise generated file was added to CVS (at the time), and have personally seen the annoyances that happen when

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
There are 3 issues: 1) mkdir -p isn't portable and so mkinstalldirs is needed 2) mkinstalldirs is distributed in /usr/lib/rpm 3) mkinstalldirs is copied and tracked in git or other VCS My comment was wrto issue #3: mkinstalldirs was originally added to RPM in order to assuage the bosses

Re: [Rpm-maint] [rpm-software-management/rpm] update mkinstalldirs to latest version (2009) (#275)

2017-07-24 Thread Jeff Johnson
You might just as easily drop mkinstalldirs. The original motivation for including mkinstalldirs was from last century: certain ancient SysV non-linux platforms lacked "mkdir -p". Carrying mkinstalldirs was the solution. -- You are receiving this because you are subscribed to this thread.

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-22 Thread Jeff Johnson
Other nitpicks in the geeky error message: the check has nothing to do with a "tag number mismatch", and lacks an "il" identifier that might have helped diagnose the problem slightly more rapidly. OTOH, I'm not sure anyone can diagnose a problem in the binary header format without a

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-22 Thread Jeff Johnson
The RPM+UUID patch in issue #269 illustrates another, more subtle usage feature of RPM_CHECK_LIB. There are two uuid.h files on a linux system: one from ossp-uuid, the other from ext4. Using --with-uuid=system (rather than --with-uuid=external) finds the correct uuid.h. Think about the

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-22 Thread Jeff Johnson
btw, this command is sufficient to identify the rpm version that built the package: `rpm -qp --qf '%{rpm version}\n' intel-aero-repo-1.4-r0.corei7_64.rpm` It looks the arguments passed used in the exact_size check are incorrect (the "ril" and "rdl" values in the error message seem wrong).

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-20 Thread Jeff Johnson
The often stated public policy here with RPM4 is "Compatibility with RPM5 is a negative goal." Don't hold your breath for a patch: I will generate and distribute a patch as soon as I understand the full consequences of recent changes to the RPM format. Yes, Yocto issues are well known. The

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-07-20 Thread Jeff Johnson
That is a package produced by RPM5: ``` $ /opt/local/bin/rpm -qp --yaml ./capsule-01.00.12-r0.core2_64.rpm | grep Rpmversion Rpmversion: 5.4.16 ``` The package uses a non-repudiable signature and verifies fine: ``` $ /opt/local/bin/rpm -qvvp ./capsule-01.00.12-r0.core2_64.rpm ... D: PUB:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-07-20 Thread Jeff Johnson
[rpm_uuid.patch.gz](https://github.com/rpm-software-management/rpm/files/1163437/rpm_uuid.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-07-20 Thread Jeff Johnson
UUID's provide a common format for identification and database retrieval. The attached patch adds a header tag format to RPM queries. - UUIDv1 time stamps (for events like build/install times): ``` $ ./rpm -q --qf '%{buildtime:uuidv1}\n' bash c60fd500-bc0a-11e7-804e-003048b801de ``` - UUIDv3

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

2017-07-19 Thread Jeff Johnson
Hmmm ... endianness *does* become an issue when a int64 is somehow returned through an int32: LE will Just Work (for small offers) and BE will fail through truncation. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

2017-07-19 Thread Jeff Johnson
Either set _romiodebug = -1, or (even better) fish out the change to set the FD_t specific debugging flag in cvtfmode() if Fopen(path, "r?.fpio") is the invocation. There's really no other way to set FD_t specific debugging because FD_t is totally opaque in rpmio/rpmio.c -- You are receiving

Re: [Rpm-maint] [rpm-software-management/rpm] [ppc64|s390x] 299: basic rpmio FAILED (rpmpython.at:35) (#259)

2017-07-19 Thread Jeff Johnson
Hmmm ... its a reasonable guess that failures on BE platforms might be related to endianness, but damfino how endianness becomes an issue on an octet oriented I/O API thru python bindings. I's suggest looking closely at the return codes from *seek/*tell which have never been well supported in

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-07-19 Thread Jeff Johnson
A (lightly tested) RPM+ZSTD patch. You will need RPM_CHECK_LIB() from the acinclude.m4 comment #2 on issue #255. [rpm_zstd.patch.gz](https://github.com/rpm-software-management/rpm/files/1160163/rpm_zstd.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-18 Thread Jeff Johnson
Here is a moderately complex real world example of how to use: detecting ZSTD either externally or internally. In configure.ac: ``` dnl # Facebook Zstd RPM_CHECK_LIB( [Facebook Zstd], [zstd], [zstd], [ZSTD_versionNumber], [zstd.h], [no,external:internal:none], [zstd:lib:lib], [

Re: [Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-18 Thread Jeff Johnson
The RPM_CHECK_LIB macro: add to acinclude.m4. [rpm_check_lib.m4.gz](https://github.com/rpm-software-management/rpm/files/1156106/rpm_check_lib.m4.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] [rpm-software-management/rpm] Add RPM_CHECK_LIB to acinclude.m4 (#257)

2017-07-18 Thread Jeff Johnson
Projects that use autoconf are expected to detect build prerequisites and libraries. This usually leads to a set of ad hoc de facto m4 macros that deal with various details like API and path incompatibilities, much of which is platform and os and distro dependent. Various standard tools (like

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-07-18 Thread Jeff Johnson
Here are URL's for zstd, brotli, and lzfse" - [https://github.com/facebook/zstd.git](url) - [https://github.com/google/brotli.git](url) - [https://github.com/lzfse/lzfse.git](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on

[Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-07-18 Thread Jeff Johnson
There are (at least) 3 new FSE compression formats that have been released in the past few years: zstd (Facebook), brotli (Google), lzfse (Apple). The new compressors have better compression ratios and speed than gzip/zlib. And while then compression rations is still not as good as XZ/LZMA, the

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

2017-07-14 Thread Jeff Johnson
I'm happy it works fine for SuSE: I'm sure you have tested. Meanwhile ^C exiting and calling rpmdbClose is a problem that needs to be solved without convoluted BDB configuration changes. Its tiresome to continuously revisit this issue, in patches and bug reports. -- You are receiving this

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

2017-07-14 Thread Jeff Johnson
@mischroe: DB_PRIVATE has its own associated durability/signal problems, including that ^C is masked while installing, and with the transaction lock being for exclusive writers, not readers. Truly ^C handling needs to be solved intelligently, not with convoluted contortions to BDB

Re: [Rpm-maint] [rpm-software-management/rpm] Convert output to the current locale (#250)

2017-07-13 Thread Jeff Johnson
The more modern interface is iconv(3). Here is the basic algorithm. The right implementation adds what is known as a header format extension to convert the encoding of any output displayed by rpm --query (aka headerSprintf() ...). hth

Re: [Rpm-maint] [rpm-software-management/rpm] Don't modify interupts if db is read only (#251)

2017-07-13 Thread Jeff Johnson
Sigh ... there are much better fixes for ^C handling than reopening an rpmdb (where ^C will *still* be disabled while installing). For starters, the signal handling in RPM was designed to be polled at code points where the rpmdb is known to be consistent (for a graceful exit with an rpmdbClose

[Rpm-maint] [rpm-software-management/rpm] Adding namespaces for macros (#246)

2017-07-01 Thread Jeff Johnson
Macro templating/configuration in RPM has become MUST'VE for building. Meanwhile, there are no conventions for sorting out the definitions, and the issues related to loading all macros all the time are getting unmanageable. E.g. here are the macros defined on a relatively recent Fedora box:

Re: [Rpm-maint] [rpm-software-management/rpm] Check for 'gettext' during configure script (#245)

2017-06-30 Thread Jeff Johnson
Did you build with --disable-nls? There is no way for RPM configure to guess whether uninstalled get text is bug of a feature: YMMV, everyone's does. ``` ./configure --help | grep nls --disable-nls do not use Native Language Support ``` -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Pre-validate (all) header data for semantic consistency (#242)

2017-06-28 Thread Jeff Johnson
As long as --nosignature and --nodigest are commonly used everywhere -- including in depsolvers -- you can setup an internal table to associate the data type with the tag and never access the field in the header. -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] Adding/moving RPMTAG_PUBKEYS to the signature header (#182)

2017-06-28 Thread Jeff Johnson
The suggestion of moving RPMTAG_PUBKEYS was directed to rpm.org (and you) as a better design for *.rpm packages, particularly if pursuing multiple signatures on multiple plaintext within an RPM package. *shrug* -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] Check signatures in the code (#240)

2017-06-22 Thread Jeff Johnson
Signatures are stored in a tag in a package header. So one retrieves a tag from a header, and then parses out the keyid from the OpenPGP format. The harder issue is that there are up to 4 (or perhaps more) possible tags where signatures might be stored, and that even if a signature with a keyid

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-13 Thread Jeff Johnson
@smorad: db-4.8.30 SHOULD be fine (but there really isn't any reason not to upgrade to more recent versions on OS X: I have direct personal/devel experience on OS X with all versions of BDB since db-4.5.x, been there, done that, WORKSFORME). -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-12 Thread Jeff Johnson
After looking at current rpm.org sources, RPM still initializes with ts->dbmode = DB_RDONLY, and then reopens the rpmdb with O_RW in rpmtsSetup(). That opens a (minor and mostly irrelevant wrto RPM on OS X) race window. There is still the ability to disable fsync (or replace with fdatasync)

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-12 Thread Jeff Johnson
Here are the patches in libdb-5.3.28-20: the patches are trackable to bug reports and appear to have been fixed by "upstream" `# License clarification patch # http://devel.trisquel.info/gitweb/?p=package-helpers.git;a=blob;f=helpers/DA

Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)

2017-06-10 Thread Jeff Johnson
You need to make sure that RPMTAG_BUILD{TIME,HOST} in both the *.src.rpm and the binary packages is identical. This was done originally with static variables, but can also be done by setting these variables in one place once, and then using the stored values where necessary. -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-10 Thread Jeff Johnson
Note also that simply removing __db.00N files to "fix" problems can lead to other "corruption" problems, particularly when other accesses are running simultaneously. The better fix for what you are calling "corruption" is to use "db_recover" with/without the -e option. BDB has a notification

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-10 Thread Jeff Johnson
The term "corruption" is highly unspecific: please avoid, and/or supply more specific details. Yes, somewhat counter intuitively, the "read-only" operation of rpm -qa is writing to disk: at a minimum, shared read locks (and rpm-4.x.y also is trying to create an advisory file lock). The

Re: [Rpm-maint] [rpm-software-management/rpm] Colorized rpmlog() output (jbj, #198) (#225)

2017-06-02 Thread Jeff Johnson
n3npq commented on this pull request. > @@ -237,6 +237,89 @@ const char * rpmlogLevelPrefix(rpmlogLvl pri) return prefix; } +static const char * rpmlogLevelColor(rpmlogLvl pri) +{ +return rpmlogMsgPrefixColor[pri&0x7]; +} + +static int rpmlogDefault(FILE *stdlog, rpmlogRec rec) +{ +

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Jeff Johnson
You need to handle the static buildtime/buildhost more carefully: the intent with build time was to have one constant and consistent timestamp everywhere during the build no matter where accessed (first access timestamps the entire build). Similarly, the build host (which is supposed to be the

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Jeff Johnson
I'm definitely interested in the patches @RPM5 (even if rather uninterested in a NSPR thread pool: I believe there are better alternatives, like OMP: YMMV). Let me look at what was needed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view

Re: [Rpm-maint] [rpm-software-management/rpm] Add comments for SHA-2 file digest variants (#223)

2017-05-30 Thread Jeff Johnson
Checking libgcrypt-1.7.6 ... yes 11 is the GnuPG assignment. `$ find . -type f -exec grep SHA224 {} \; * gcrypt.h (GCRY_MD_SHA224): Change value from 306 to 11 to match * gcrypt.h: Add GCRY_MD_SHA224. GCRY_MD_SHA224 = 11, ` -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Add comments for SHA-2 file digest variants (#223)

2017-05-30 Thread Jeff Johnson
RFC 4880 changes too slowly to add new algorithms. Tracking GNUPG is likely faster/saner. FWIW, AFAIK (I wouldn't have added to the "official" OpenPGP space if not so) gnupg has assigned 11 to SHA224: `rpmiotypes.h:PGPHASHALGO_SHA224= 11, /*!< SHA-224 */` -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-24 Thread Jeff Johnson
Again ... OPENMP is the least intrusive (but perhaps the most abstract) implementation. OPENMP has queues and pools and is viable even in 2017. I parallelized the rpm --verify loop over files a few years back using OPENMP. Here are the top level annotations that were needed: `#if

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-03 Thread Jeff Johnson
Yes its quite feasible to package binary *.rpm files in parallel. The hardest problem is choosing a framework for parallelization: OMP, or pthreads or (in RPM5) yarn. A general framework for parallelization is needed, with queues and thread pools and output serialization etc, is needed because

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-19 Thread Jeff Johnson
Yes, fdatasync+fadvise_syncfs removes pages from system cache: ` --> Fincore(0x61408e40) fdno 16 path /opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f7761a <-- Fincore(0x61408e40) fdno 16 rc 0 <-- fdClose(0x61408e40) rc 0| UFD -1 fp 0x61204e40

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-18 Thread Jeff Johnson
@cgwalters: easy enough to wrap syncfs(2) into a measurement harness: ` ==> Fclose(0x61408e40) | LIBIO 0x61204fc0(-1) fdno -1 | UFD 16 fp 0x61204fc0 --> Fincore(0x61408e40) fdno 16 path /opt/local/tmp/lib/modules/4.11.0-0.rc6.git2.1.fc27.x86_64/System.map;58f6dc66 <--

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-16 Thread Jeff Johnson
@pmatila: if you *do* attempt to configure +/- fsync(2) somehow, then durability is likely the better approach. By "durability", I mean "this file is essential to booting" (and hence is worth paying the cost of fsync(2) to ensure that the file is correctly written to disk). Some simple

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-16 Thread Jeff Johnson
Wall clock time measurements for an RPM command (this is RPM5, but the I/O trace is identical to rpm.org): `sudo /usr/bin/time /opt/local/bin/rpm -Uvh --nodeps --noscripts --relocate /=/opt/local/tmp /var/tmp/kernel-core-4.11.0-0.rc6.git2.1.fc27.x86_64.rpm` BASE 2.26user 0.43system

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
Yes, with about 20 other similar articles and implementations, some even accurate. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
The buffer is 32 * BUFSIZ )= 128Kb) not 32Kb. Sorry for the confusion. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
One last note: doing fdatasync once, before fadvise+fsync, not after every write is ~10x faster: ` FDIO: 30 writes, 3763601 total bytes in 0.003558 secs FDIO: 1 allocs, 3763601 total bytes in 0.24 secs FDIO: 1 dsyncs,0 total bytes in 0.074081 secs FDIO: 1

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-14 Thread Jeff Johnson
If you are arguing for "opt-in" rather than "opt-out" by default when a macro is not explicitly set, then change the "1" in this line: ` _periodic_fsync = rpmExpandNumeric("%{?_periodic_fsync}%{!?_periodic_fsync:1}"); ` Otherwise set %_periodic_fsync as you wish. There wasn't

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-13 Thread Jeff Johnson
A measurement of the impact is needed first, enabler/disabler configuration can always be added. In general, RPM does best with "opt-out", not "opt-in" policies. There may be other changes, such as doing the fdatasync() less often, may be useful too. -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: option for initial-only content in files (#152)

2017-04-12 Thread Jeff Johnson
Add * update by patching and make the policies platform and machine specific. Then call the implementation something catchier than ansible or DevOps. Seriously: configuration != package management. -- You are receiving this because you commented. Reply to this email directly or view it on

[Rpm-maint] [rpm-software-management/rpm] Colorized rpmlog() output (+PATCH) (#198)

2017-04-11 Thread Jeff Johnson
SSIA Have fun! [rpm+color.patch.gz](https://github.com/rpm-software-management/rpm/files/915427/rpm.color.patch.gz) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding high precision event time stamps to RPM (#197)

2017-04-10 Thread Jeff Johnson
Attached is patch to extend RPMTAG_{BUILDTIME,INSTALLTIME,INSTALLTID} to add a 2nd element that contains a struct timespec tspec.tv_nsec 2nd element. There are 2 new getters/setters added to the rpmts API: rpm{Get,Set}Tid2(). About the only subtlety (so far) is the need (for rpmdb "legacy

Re: [Rpm-maint] [rpm-software-management/rpm] multiple signatures support? (#189)

2017-04-09 Thread Jeff Johnson
There is nothing stopping other applications from using detached signatures on *.rpm files as necessary. RPM cannot carry one (or multiple) signatures within signed plaintext. (aside) In principle a different ping-pong like signing could be attempted to ensure that both signature/metadata

Re: [Rpm-maint] [rpm-software-management/rpm] Make 'rpm -V' more resistent against rpmdb manipulations (#196)

2017-04-09 Thread Jeff Johnson
Hmmm ... its not clear what exploit is used (from just reading the file at the URL you gave). The RPM versions (4.1/4.2) mentioned in the URL are quite ancient (circa RHL 8/9). Even on those ancient systems, /var/lib/rpm requires access to root:root (or rpm:rpm) in order to modify an rpmdb

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-09 Thread Jeff Johnson
Here is an updated patch, including the bare minimum changes with a %_periodic_fsync "opt-out" macro to the "production" rpmfiArchiveReadToFilePsm() to use fdatasync+posix_fadvise+fsync. I've left all the debugging gook in the patch to document the --rpmiodebug trace and to illustrate the

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-09 Thread Jeff Johnson
OK, here is one way to install a file without leaving the file in the cache. The simplest explanation is to show the --rpmiodebug log (interspersed with some debugging): *** Fopen ufdio path /usr/share/doc/hello-2.0/README;58ea9807 fmode w+.ufdio ***

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-08 Thread Jeff Johnson
I knew if I wrote a patch, better answers would instantly appear. There is fadvise(FADV_DONTNEED) wrto fsync(2) analogous to madvise(MADV_DONTNEED) wrto msync(2). I will add to my patch in the next few days for measurement, todo++. The easy answer for your "Chef installs packages and my

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-08 Thread Jeff Johnson
As promised, here is a proof-of-concept patch to permit using either fwrite+fsync or mmap+madvise+msync while writing files onto the file system from a *.rpm package. The patch passes "make check" (with the current known failures) on tip. There are various switches to enable/disable the added

[Rpm-maint] [rpm-software-management/rpm] Print exit code with rpm -vv (#195)

2017-04-08 Thread Jeff Johnson
This trivial patch helps diagnose rpm behavior while developing or testing. ` diff --git a/rpmqv.c b/rpmqv.c index 8655776..4f448c2 100644 --- a/rpmqv.c +++ b/rpmqv.c @@ -354,5 +354,7 @@ int main(int argc, char *argv[]) rpmcliFini(optCon); +rpmlog(RPMLOG_DEBUG, "exit code: %d\n",

Re: [Rpm-maint] [rpm-software-management/rpm] ELF file conflict 'color' resolution for tri-lib systems (#193)

2017-04-08 Thread Jeff Johnson
Resolving file "colors" affects more than MIPS64 n32: the x32 format also uses (1 <<2) color bit and (potentially) needs RPM to choose between executables that are compiled differently. AFAIK, PLD has succeeded in adding support for x32 to RPM5, if you need patches that have been tested in

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-08 Thread Jeff Johnson
Thanks for measurements! Could you also attempt to measure the effect on RPM too? Wall clock time is sufficient, as the time necessary to perform an upgrade is what most RPM users are sensitive to. Your graphs (presumably on SSD?) show smaller/periodic writes w fsync enabled, and larger/delayed

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-06 Thread Jeff Johnson
@jayzmh: major invasive change to ... RPM? or something else? I'll happily write the code for rpm.org RPM if you wish. I fail to see how reverting to what RPM used to do (i.e. use mmap with MADV_DONTNEED and most importantly for you using msync(MS_ASYNC) which schedules flush to disk like

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-06 Thread Jeff Johnson
@jayzmh - add a static variable to expand the macro as a "one shot" ... there are many places in rpm that do that, hurts little (until rpm can capture global configuration object). Meanwhile I recommend the *_DONTNEED mmap implementation instead of fsync. RPM used that for years, and

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
There is a modestly obscure means to attach *_DONTNEED (to achieve kernel delayed writes and removing pages when I/O is complete) using the existing rpm.org rpmfiArchiveReadToFilePsm() Fwrite->write(3) loop calling setvbuf(3) immediately after Fopen on the fopencookie(3) FILE *. (aside)

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
Hmmm ... glibc (which uses mmap(2) for stdio if requested) would benefit from being able to mark the stdio buffer-in-use with *_DONTNEED using madvise/posix_fadvise. At a minimum, cvtfmode() in rpmio/rpmio.c would need to be extended to pass the glibc-peculier 'm' flag through to fopen(3)

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
For completeness (and humor ;-), there is also O_DIRECT. See [https://lkml.org/lkml/2007/1/10/233](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
Another issue is that certain file systems (NFS? DFS? I forget ...) also delay I/O operations until close(2). What I recall is that an application MUST check the return code of close(2) in order to ensure that the I/O was actually performed: it isn't sufficient to check the return code of

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
There is an alternative (but more complicated) lazy implementation that might be attempted by doing a fsync in Fwrite (or fdWrite) when the accumulated I/O passes a certain mark. H ... A general solution to add a resource cap on all rpmio I/O could also be attempted (but isn't easy).

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-05 Thread Jeff Johnson
FYI: several notes (after looking at the patch) 1) The measurements I cited above are using mmap(2) to decompress directly into pages (avoiding a buffer copy) marked with MADV_DONTNEED (which permits delayed/larger writes) to avoid a buffer copy. In addition, there is a check on file size of

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-04 Thread Jeff Johnson
Good if doing in fsm.h ( I haven't looked). Meanwhile a 50x slower installation is intolerable for a package manager even if opt-in. Nor can an application like RPM be expected to detect whether the underlying filesystem is on SSD or USB or rotational disk and adapt accordingly: there simply

Re: [Rpm-maint] [rpm-software-management/rpm] Add macro to force fsync() on close() (#187)

2017-04-04 Thread Jeff Johnson
Do you have measurements of the effect of fsync()? The last time sync/fdatasync were suggested for RPM that I am aware of was ~2011 in Mandriva. The issue comes up every few years this century ... Here was the effect of adding fsync/fdatasync to RPM, measure on otherwise identical system with

Re: [Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

2017-04-02 Thread Jeff Johnson
(answering my own question) Short answer: digesting compressed payload instead of compressing the digested payload is expedient. Long answer: RPM goes to some lengths to preserve streaming-without-peeking-at-magic for *.rpm packages. The usual RPM approach is/was to add/save tags in the

[Rpm-maint] [rpm-software-management/rpm] Q: RPMTAG_PAYLOADDIGEST: compute before or after compression? (#184)

2017-03-28 Thread Jeff Johnson
The recently added RPMTAG_PAYLOADDIGEST computes a digest on the compressed payload. Perhaps now is the time to change the plaintext used to be the uncompressed rather than the compressed archive payload? The benefit of the digest on the uncompressed archive is that the digest becomes immune

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

2017-03-28 Thread Jeff Johnson
Ah, you have added RPMSIGTAG_RESERVEDSPACE already which I was not expecting. The (minor but annoying problems) with the assignment to 1008 is that RPMSIGTAG_RESERVEDSPACE collides with RPMTAG_INSTALLTIME, and will not sort correctly to the end of an immutable region data section as a "free

Re: [Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

2017-03-28 Thread Jeff Johnson
My bad for not noticing. Good that padding has been added by adding (apparently) 0x1000 to the data size: `0060: 8ead e801 0006 1074 ...t` Adding a "free space" tag (as @rpm5.org) provides an indication that the signature header is padded. Meanwhile, the

[Rpm-maint] [rpm-software-management/rpm] Reserving space in the signature header (#183)

2017-03-27 Thread Jeff Johnson
The signature header contains (optional) information to verify package integrity. So new tags are added when packages are signed, and a new signature header is created and inserted into a *.rpm package file. However, the insertion forces a rewrite of the metadata header and payload, which for

[Rpm-maint] [rpm-software-management/rpm] Adding/moving RPMTAG_PUBKEYS to the signature header (#182)

2017-03-27 Thread Jeff Johnson
RPM package files can include public keys in the metadata header. So in principle, a package can include the public key used to sign a package and verify the package signature when reading package headers. The RPMTAG_PUBKEYS array is also within the signed immutable region and cannot be

[Rpm-maint] [rpm-software-management/rpm] Tags in the signature header are not sorted (#181)

2017-03-25 Thread Jeff Johnson
This is the hex dump of the createrepo_c Archer-3.4.5-6.x86_64.rpm signature header. RPMSIGTAG_SHA1 follows RPMSIGTAG_MD5 at offset 0xa0, forcing a sort while loading the signature header. _ 0060: 8ead e801 0005 0054 ...T 0070: 003e 0007

[Rpm-maint] [rpm-software-management/rpm] Retrofitting a format version on *.rpm packaging. (#172)

2017-03-07 Thread Jeff Johnson
When originally written, the rpmlead contained information used to identify a *.rpm. Twenty years later, most of the rpmlead information is de facto and ad hoc and historical. Progress using other information, like rpmlib(...) tracking dependencies, has managed to associate new "features"

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

2017-03-06 Thread Jeff Johnson
Another issue that might make debugedit.c more portable is similar to inlining DW_FORM* instead of `#include ` Instead of using EU version dependent #defines for strtab_init() etc (which needs -lebl), it may be more "portable" to just inline the routines and maintain a copy in debugedit.c.

Re: [Rpm-maint] [rpm-software-management/rpm] debugedit.c patches (#171)

2017-03-06 Thread Jeff Johnson
Yes all the patches were reversed. Meanwhile I just sorted the first patch with Mark Hatle (aka "fray"). The segfault is not reproducible with EU-0.168. The hex printing was an attempt to avoid pgpHexStr() which differs between rpm.org and rpm5.org. You explained away the valid_file() check

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
debugedit.c --help can be improved (and potentially internationalized) *shrug* @@ -1323,16 +1290,15 @@ static struct poptOption optionsTable[] = { { "base-dir", 'b', POPT_ARG_STRING, _dir, 0, -N_("base build DIRECTORY of objects"), N_("DIRECTORY") }, + "base build directory of

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
Portability of debugedit.c on systems that do not support #include (solaris, *BSD) can be achieved by duplicating certain DWARF standard constants: -#ifndef EM_AARCH64 -#define EM_AARCH64 183 /* ARM AARCH64 */ -#endif -#ifndef R_AARCH64_ABS32 -#define R_AARCH64_ABS32 258

Re: [Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
For rpm portability with older versions of elfutils, this harmless sanity check might be useful: (excuse the different hash functions in use in the patch snippet below) @@ -1533,40 +1493,12 @@ /* Now format the build ID bits in hex to print out. */ { const uint8_t * id = (uint8_t

[Rpm-maint] [rpm-software-management/rpm] debug edit build generation is endian dependent when cross-compiling? (#171)

2017-03-05 Thread Jeff Johnson
rpm5.org carries this patch from Yocto: https://patchwork.openembedded.org/patch/46887 which likely should be added when updating debug edit.c soon. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

[Rpm-maint] Keep my name out of your discussions

2010-06-21 Thread Jeff Johnson
Re: BTW, Jeff J. must have changed his mind at some point, because rpm5 allows short-circuiting of -bb. You have no right to discuss the state of my mind. In fact, you haven't a clue. Keep my name out of your gossipy garbage. 73 de Jeff ___

Re: [Rpm-maint] relicensing for Maximum RPM

2009-09-07 Thread Jeff Johnson
of the ones who worked for rht at the time they were writing it. So I suspect Paul and jbj's contributions are (c) Red Hat inc. But Matthias and Ville are (c) themselves, unless otherwise disclaimed. You may suspect all you wish, but you better check dates before making false claims in

<    2   3   4   5   6   7