Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-07-30 Thread Jeff Johnson
@ignatenkobrain : I don't send PR's because I don't have the time/energy/resources to wait months for review, updating the PR with zero feedback other than "Why don't you send PR's?". That's why. -- You are receiving this because you are subscribed to this thread. Reply to this email directly

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-28 Thread Jeff Johnson
In order to run queries as non-root, there are three minor changes: 1) lock.mdb (an 8Kb file) needs to be 0666, other perms as expected. 2) don't bother retrying mdb_env_open(): I added this line to display errors as well: ``` rpmlog(RPMLOG_ERR, "lmdb: %s(%s/lock.mdb): %s\n", __FUNCTION__,

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-28 Thread Jeff Johnson
One last comparison: RPM5 uses BerkeleyDB too, but with different tunables than RPM4. Here's essentially the same measurement as the previous comment: ``` 1.15user 0.87system 0:02.11elapsed 96%CPU (0avgtext+0avgdata 584900maxresident)k 240inputs+390776outputs (7major+355303minor)pagefaults

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-07-30 Thread Jeff Johnson
See also comments at issue #281 about removing MDB_FIXEDMAP. Just delete the line that looks like `eflags |= MDB_FIXEDMAP` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-07-30 Thread Jeff Johnson
BDB and LMDB have always had the ability to have mutable databases (actually tables) in a single file. So actually there is a "Packages" (sub-)database/table in data.mdb. Since 2 of 3 backend now have sub-databases in a single file, its likely time to do the same with BDB. I've never bothered

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-07-30 Thread Jeff Johnson
@Conan-Kudo : if using RPM_CHECK_LIB w internal, its a bit trickier than just adding a checkout. So you likely want to remove "internal" as the 1st thing to try in "[yes,internal:external:none]" FYI, removing "none" as in (yes,external) will fail to configure if not found (i.e. mandatory) And

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-28 Thread Jeff Johnson
For completeness, here is the same measurement with RPM+NDB: ``` $ /usr/bin/time ./rpm -qa --ndb > /dev/null 3.25user 0.13system 0:03.38elapsed 100%CPU (0avgtext+0avgdata 17664maxresident)k 0inputs+0outputs (0major+8164minor)pagefaults 0swaps ``` -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Add zstd compressed payload support to RPM (from @n3npq in #256) (#295)

2017-07-31 Thread Jeff Johnson
n3npq commented on this pull request. > @@ -203,6 +204,20 @@ AC_CHECK_HEADERS([lzma.h],[ AC_SUBST(WITH_LZMA_LIB) #= +# Check for zstd library. + +dnl # Facebook Zstd +RPM_CHECK_LIB( RPM_CHECK_LIB uses pkgconfig (when available). I'm not sure what you wish. -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Add zstd compressed payload support to RPM (from @n3npq in #256) (#295)

2017-07-31 Thread Jeff Johnson
n3npq commented on this pull request. > @@ -348,6 +348,12 @@ static char *getIOFlags(Package pkg) compr = "lzma"; (void) rpmlibNeedsFeature(pkg, "PayloadIsLzma", "4.4.6-1"); #endif +#if HAVE_ZSTD_H + } else if (rstreq(s+1, "zstdio")) { + compr = "zstd";

Re: [Rpm-maint] [rpm-software-management/rpm] Add zstd compressed payload support to RPM (from @n3npq in #256) (#295)

2017-07-31 Thread Jeff Johnson
n3npq commented on this pull request. > @@ -348,6 +348,12 @@ static char *getIOFlags(Package pkg) compr = "lzma"; (void) rpmlibNeedsFeature(pkg, "PayloadIsLzma", "4.4.6-1"); #endif +#if HAVE_ZSTD_H + } else if (rstreq(s+1, "zstdio")) { + compr = "zstd";

Re: [Rpm-maint] [rpm-software-management/rpm] Add zstd compressed payload support to RPM (from @n3npq in #256) (#295)

2017-07-31 Thread Jeff Johnson
n3npq commented on this pull request. > @@ -1278,6 +1278,11 @@ static const struct rpmlibProvides_s rpmlibProvides[] > = { { "rpmlib(RichDependencies)","4.12.0-1", ( RPMSENSE_EQUAL), N_("support for rich dependencies.") }, +#if HAVE_ZSTD_H +{

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-29 Thread Jeff Johnson
After adding equivalent disabling of fsync and its ilk to mdb_env_open(): ``` eflags |= MDB_FIXEDMAP; #ifndef HACK eflags |= MDB_WRITEMAP; eflags |= MDB_NOMETASYNC; eflags |= MDB_MAPASYNC; eflags |= MDB_NOSYNC; eflags |= MDB_NOTLS; eflags |=

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-07-29 Thread Jeff Johnson
Ah, I missed this comment in lmdb.h (and the order of mdb_txn_commit() and mdb_cursor_close() are incorrectly reversed in lmdb_dbiCursorFree()) ``` /** @brief Close a cursor handle. * * The cursor handle will be freed and must not be used again after this call.

Re: [Rpm-maint] [rpm-software-management/rpm] rpmio: ifdef out lzma and bzip variable (33ca6d0)

2017-08-09 Thread Jeff Johnson
HAVE_ZSTD should be HAVE_ZSTD_H -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] rpmio: ifdef out lzma and bzip variable (33ca6d0)

2017-08-09 Thread Jeff Johnson
You can certainly use any Baroque'n logic scheme you wish. What is done in the check-in is condition BZIP on a test for an include, while ZSTD is conditioned on a test for a library. Consistency is important too. -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-07 Thread Jeff Johnson
@cgwalters when moving 100's of megabytes through a finite resource, there will be a measurable effect on the cache. that's true for any large I/O operation including rpm-ostree: have you run mincore to see how many pages are left in cache after syncfs? typically syncfs/sync guarantee write to

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-08-06 Thread Jeff Johnson
The rpm_uuid.patch is now applied to an RPM tree at [https://github.com/rpm5/rpm/commit/c8c72fb195790dfa2d40c20ca597d9cc2b75](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
@Conan-Kudo I do have something to add to RPM... filesystem durability and preventing cache blowout at modest cost. Not everything is a "patch". -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
@ignatenkobrain: send a PR ... for what? This work is an improvement on PR #197 which hasn't moved in 6+ months. And -- as stated -- the patches here are "rude-and-crude" and necessary to do the justification for adding an event loop to RPM. There is nothing here that should be added to RPM

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-08-06 Thread Jeff Johnson
The final patch (with minor changes) is is now committed at https://github.com/rpm5/rpm/commit/528925bd9967d9b741787df56e46a5c83c7a54aa -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
The libeio event loop adds RPM_CHECK_LIB to handle --with-libeio here: [https://github.com/rpm5/rpm/commit/133c60b4036fd97af89390aea6ae94034239d545](url) (Presumably you will do your own pkg-config AutoFU to handle --with-libeio) The patch in comments above has been mildly reworked to remove

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-08-06 Thread Jeff Johnson
This patch is now committed at https://github.com/rpm5/rpm/commit/debde46a23c5a942d017026903020eb3b389d91c -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
FWIW, here is the rather pleasant and complete documentation for libeio: [http://pod.tst.eu/http://cvs.schmorp.de/libeio/eio.pod](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: extract changelog to file, include as %doc automatically and put stub in actual binary and source RPMs. (#305)

2017-08-18 Thread Jeff Johnson
Um ... its not that simple, and the idea of mapping out changelings to a file that is then retrofitted to "rpm --changelog" pretends to a silly legacy compatibility (and adds complexity for not much gain other than not having to justify the change). You are more than a little late to the

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: extract changelog to file, include as %doc automatically and put stub in actual binary and source RPMs. (#305)

2017-08-19 Thread Jeff Johnson
The best approach to making changelogs "available before installing" is to make the changelings available before downloading. This is already being done with security fixes that supply relevant info for mature distributions. Basically resurrect the rpm-metadata XML/SQLITE formats and distribute

Re: [Rpm-maint] [rpm-software-management/rpm] Static code analysis task for make (#306)

2017-08-19 Thread Jeff Johnson
You can do style checks if you wish. Meanwhile RPM is/was permeated with false positives like ``` foo = _free(foo); ``` which is my own personal idiolect of C: wipe oy the pointer when you free the memory. Huge PITA handling in Coverity: huge number of warnings about "set but unused

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: extract changelog to file, include as %doc automatically and put stub in actual binary and source RPMs. (#305)

2017-08-19 Thread Jeff Johnson
No argument re "shrink" whatsoever. Meanwhile no one has ever written the statistics gathering program, its all just "IMO ..." I absolutely know that an alternative representation for SHA256 file digests encoded in hex strings would be (at least) an order of magnitude larger reduction in the

Re: [Rpm-maint] [rpm-software-management/rpm] Remove redundant max_macro_depth initialization (2986e76)

2017-08-19 Thread Jeff Johnson
W00t! -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/commit/2986e7659ddac012e8d7c6c57fbc04aaa3f4c84f#commitcomment-23748729___

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: extract changelog to file, include as %doc automatically and put stub in actual binary and source RPMs. (#305)

2017-08-19 Thread Jeff Johnson
And I do point out GIGO ... its entirely within your power to use existing implementations in RPM to get rid of changeling perceived "bloat". RPM's job is to ensure flawless delivery from build -> install machine in spite of everyone's opinions. Go for it! -- You are receiving this because

Re: [Rpm-maint] [rpm-software-management/rpm] RFC: give some freshness to brp scripts (#308)

2017-08-22 Thread Jeff Johnson
I think you need to describe a more complete example of what you think needs additional "freshness" in build root policy scripts: the need cannot be deduced from the terse descriptions of problems in comment #1. -- You are receiving this because you are subscribed to this thread. Reply to this

Re: [Rpm-maint] [rpm-software-management/rpm] RFC: standard macro syntax for macro files (#309)

2017-08-22 Thread Jeff Johnson
You need to provide better examples, not statements of deficiencies with "macro files" (which are presumably the files where RPM configuration is defined). In some sense you are asking a "library" (i.e. macro files) to be able to reconfigure to satisfy "application" (i.e. rpmbuild ) needs. A

Re: [Rpm-maint] [rpm-software-management/rpm] Make macro %{error:...} cause an actual failure (94e8cd6)

2017-08-22 Thread Jeff Johnson
Um yes, sure. You realize that I implemented both %warn and %error in RPM? My comment is based on re-thinking the usage case of %warn/%error this century. Fail on %error if you wish. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-08-17 Thread Jeff Johnson
Code duplication (and refactoring) are the least of the problems with the current RPM backend. There is no concept of a "transaction" or "durability" in the current RPM backend. And the existing INIT_CDB Berkeley DB model implemented has been tortured beyond belief or reason. Anything that

Re: [Rpm-maint] [rpm-software-management/rpm] add support for premacros.d, complementing --predefine (#304)

2017-08-17 Thread Jeff Johnson
Pre-macros (and --predefine) aren't the right implementations: either an RPM invocation can read its own configuration or it can't. Playing semantic games with goose-loosen CLI options adds complexity for no gain: all that is happening under the hood is that some macros are read before others.

Re: [Rpm-maint] [rpm-software-management/rpm] Add LMDB backend for RPMDB to RPM (from @n3npq in #281) (#291)

2017-08-17 Thread Jeff Johnson
FWIW, "data.mdb" can be renamed to "Packages.mdb" in LMDB by adding a flag and passing the file path, not the directory path, when opening. Not worth the effort (and adds confusion: "Packages" is the name of a database/table within "data.mdb" not the name of the file). Note that Berkeley DB

Re: [Rpm-maint] [rpm-software-management/rpm] Make macro %{error:...} cause an actual failure (94e8cd6)

2017-08-17 Thread Jeff Johnson
Causing a full stop failure during expansion raises expectations to provide better RPM error messages, or behave more gracefully. The expectations often cannot be met because macro expansion is context free (by design). Debugging a macro expansion that is exiting is often harder than printing

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-03 Thread Jeff Johnson
Yes its quite feasible to package binary *.rpm files in parallel. The hardest problem is choosing a framework for parallelization: OMP, or pthreads or (in RPM5) yarn. A general framework for parallelization is needed, with queues and thread pools and output serialization etc, is needed because

Re: [Rpm-maint] [rpm-software-management/rpm] Check signatures in the code (#240)

2017-06-22 Thread Jeff Johnson
Signatures are stored in a tag in a package header. So one retrieves a tag from a header, and then parses out the keyid from the OpenPGP format. The harder issue is that there are up to 4 (or perhaps more) possible tags where signatures might be stored, and that even if a signature with a keyid

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-05-24 Thread Jeff Johnson
Again ... OPENMP is the least intrusive (but perhaps the most abstract) implementation. OPENMP has queues and pools and is viable even in 2017. I parallelized the rpm --verify loop over files a few years back using OPENMP. Here are the top level annotations that were needed: `#if

Re: [Rpm-maint] [rpm-software-management/rpm] Add comments for SHA-2 file digest variants (#223)

2017-05-30 Thread Jeff Johnson
RFC 4880 changes too slowly to add new algorithms. Tracking GNUPG is likely faster/saner. FWIW, AFAIK (I wouldn't have added to the "official" OpenPGP space if not so) gnupg has assigned 11 to SHA224: `rpmiotypes.h:PGPHASHALGO_SHA224= 11, /*!< SHA-224 */` -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Add comments for SHA-2 file digest variants (#223)

2017-05-30 Thread Jeff Johnson
Checking libgcrypt-1.7.6 ... yes 11 is the GnuPG assignment. `$ find . -type f -exec grep SHA224 {} \; * gcrypt.h (GCRY_MD_SHA224): Change value from 306 to 11 to match * gcrypt.h: Add GCRY_MD_SHA224. GCRY_MD_SHA224 = 11, ` -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Jeff Johnson
You need to handle the static buildtime/buildhost more carefully: the intent with build time was to have one constant and consistent timestamp everywhere during the build no matter where accessed (first access timestamps the entire build). Similarly, the build host (which is supposed to be the

Re: [Rpm-maint] [rpm-software-management/rpm] Create binary rpm packages in parallel? (better multiple CPU core utilization) (#211)

2017-06-01 Thread Jeff Johnson
I'm definitely interested in the patches @RPM5 (even if rather uninterested in a NSPR thread pool: I believe there are better alternatives, like OMP: YMMV). Let me look at what was needed. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view

Re: [Rpm-maint] [rpm-software-management/rpm] Colorized rpmlog() output (jbj, #198) (#225)

2017-06-02 Thread Jeff Johnson
n3npq commented on this pull request. > @@ -237,6 +237,89 @@ const char * rpmlogLevelPrefix(rpmlogLvl pri) return prefix; } +static const char * rpmlogLevelColor(rpmlogLvl pri) +{ +return rpmlogMsgPrefixColor[pri&0x7]; +} + +static int rpmlogDefault(FILE *stdlog, rpmlogRec rec) +{ +

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-10 Thread Jeff Johnson
The term "corruption" is highly unspecific: please avoid, and/or supply more specific details. Yes, somewhat counter intuitively, the "read-only" operation of rpm -qa is writing to disk: at a minimum, shared read locks (and rpm-4.x.y also is trying to create an advisory file lock). The

Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)

2017-06-10 Thread Jeff Johnson
You need to make sure that RPMTAG_BUILD{TIME,HOST} in both the *.src.rpm and the binary packages is identical. This was done originally with static variables, but can also be done by setting these variables in one place once, and then using the stored values where necessary. -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-10 Thread Jeff Johnson
Note also that simply removing __db.00N files to "fix" problems can lead to other "corruption" problems, particularly when other accesses are running simultaneously. The better fix for what you are calling "corruption" is to use "db_recover" with/without the -e option. BDB has a notification

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-13 Thread Jeff Johnson
@smorad: db-4.8.30 SHOULD be fine (but there really isn't any reason not to upgrade to more recent versions on OS X: I have direct personal/devel experience on OS X with all versions of BDB since db-4.5.x, been there, done that, WORKSFORME). -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-12 Thread Jeff Johnson
After looking at current rpm.org sources, RPM still initializes with ts->dbmode = DB_RDONLY, and then reopens the rpmdb with O_RW in rpmtsSetup(). That opens a (minor and mostly irrelevant wrto RPM on OS X) race window. There is still the ability to disable fsync (or replace with fdatasync)

Re: [Rpm-maint] [rpm-software-management/rpm] Rpm query causes corruption in the file-backed mmaped bdb regions (#232)

2017-06-12 Thread Jeff Johnson
Here are the patches in libdb-5.3.28-20: the patches are trackable to bug reports and appear to have been fixed by "upstream" `# License clarification patch # http://devel.trisquel.info/gitweb/?p=package-helpers.git;a=blob;f=helpers/DA

Re: [Rpm-maint] [rpm-software-management/rpm] Adding/moving RPMTAG_PUBKEYS to the signature header (#182)

2017-06-28 Thread Jeff Johnson
The suggestion of moving RPMTAG_PUBKEYS was directed to rpm.org (and you) as a better design for *.rpm packages, particularly if pursuing multiple signatures on multiple plaintext within an RPM package. *shrug* -- You are receiving this because you are subscribed to this thread. Reply to

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-14 Thread Jeff Johnson
There are significant differences between the glibc and the rpm copies of fts.: they are different routines in the ABI even if they started from the same code base, there are (at least) 4 extra pointers. Arguing about "bloat" or proper use of libraries for routines that are intrinsically

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: run static code analysis in CI (#306)

2017-09-14 Thread Jeff Johnson
OK. I'll create a RPM.ORG project at scan.coverity.com and import the code to assess the level of pain (and there _WILL_ be pain: it took me several weeks of mind-numbing clickety poo to get the level of reported positives from the coverity firehose down to something that was useful for rpm

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: add support for `else if` operator (#311)

2017-09-15 Thread Jeff Johnson
The difference is that %elif needs to be manually replicated in many spec files, while a common stanza instantiated in a file can be included where needed. IOW refactoring a structural solution to some configuration problem is far easier. The logic is (of course) identical. -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Run binary package generation via thread pools (#226)

2017-08-30 Thread Jeff Johnson
@ignatenkobrain: tex (with many packages all with few files) isn't the best benchmark to illustrate a parallel packaging speedup. There are a finite number of threads in the thread pool (which you seem not to have reported or controlled for, sigh). Try a benchmark with a few packages with lots

Re: [Rpm-maint] [rpm-software-management/rpm] WIP/RFE: Hint to users to use ostree/rpm-ostree if we get EROFS (#320)

2017-09-09 Thread Jeff Johnson
@cgwalters nagware never works. if ostree permits live update even with -EROFS, then your patch should teach rpm to do similar live-update, not spew nagware adverts. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-11 Thread Jeff Johnson
@mhatle I've replaced fakechroot with pseudo its a couple of lines. Almost everything Just Works ... the few tests failing can likely be fixed rapidly. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] rpm's doFoo function tends to expand twice (#313)

2017-09-05 Thread Jeff Johnson
FWIW, %uncompress (and all the insane overhead in RPM to associate compressors/uncompressors with compressed files) is better dealt with by writing a meta-compressor in C that handles one well known set of options and Does The Right Thing. FYI: ZSTD will handle gzip/zlib, lzma/xz, and lz4

Re: [Rpm-maint] [rpm-software-management/rpm] add support for premacros.d, complementing --predefine (#304)

2017-09-06 Thread Jeff Johnson
There's surely prettier syntax that can be achieved, or alternative syntaxes similar to what make does with conditional sections. E.g. `%{!?_arch_x86_64:#}%foo bar` -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] add support for premacros.d, complementing --predefine (#304)

2017-09-06 Thread Jeff Johnson
There's another approach to conditionalize macro definitions from configuration by permitting existence tests in the first field. Most of the usage cases that have been reported to me are based on arch/os/platform, and the selection is being done by putting (some value) of arch in path to

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-13 Thread Jeff Johnson
@mhatle When I tried with pseudo, replacement was almost, but not quite, seamless and drop-in. I'll try to revisit the issues later this week ... it was more than a year ago that I tried using pseudo. -- You are receiving this because you are subscribed to this thread. Reply to this email

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-13 Thread Jeff Johnson
The other looming issue is stat(2), the new fangled stat replacement from David Howells. xstat(2) is going to require more significant changes to fts.c than LFS. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-12 Thread Jeff Johnson
I suppose I should donate copies of opendir/readdir/stat/lstat that "work" with rpm Header's. The world is littered with sloppy code trying to reconstruct stat(2) information in custom code based on headerGet(). Using fts.c to bury all the details of headerGet() is far easier to program with

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-12 Thread Jeff Johnson
If all you wish is RPM with LFS support, the changes to add LFS support are quite minor, and are better addressed as a RFE, rather than an arch claim about "fixes". Just saying ... But where is LFS support needed these days? -- You are receiving this because you are subscribed to this thread.

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-12 Thread Jeff Johnson
@ldv-alt Rpm deals with many hierarchies, including files in payloads/transactions, where fts.c is a convenient abstraction for traversal when customized opendir et al are added. Issues with current maintainer can at least be dealt with now that you have added your complaints. -- You are

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Revert "Only build bundled fts if system has a bad version th… (#324)

2017-09-12 Thread Jeff Johnson
The ability to replace opendir/readdir/stat/lstat permitted walking remote URI's and are otherwise quite obvious changes. Go ahead and show me how to use the glibc fts.c to walk remote URI's. The other reason for internalizing fts.c was an incompatible change to how symlinks were traversed

Re: [Rpm-maint] [rpm-software-management/rpm] Sync disks at the end of transactions (RhBug:1461765) (b7a869f)

2017-09-06 Thread Jeff Johnson
See issue #258 for faster (and better: includes cache invalidation) than sync(2). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Proposal for default init.lua (#321)

2017-09-25 Thread Jeff Johnson
Everyone wants modules: rpm needs to support both LPATH and CPATH, in a private rpm tree as well as "standard" system paths. init.lua "support" has been in place since 2003 and no-one (that I know of) has made any serious attempt to use. -- You are receiving this because you are subscribed to

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Automatic (sub)package generators (#329)

2017-10-03 Thread Jeff Johnson
FWIW, automating sub packages was considered by Toby in Unity linux back in 2009 (until there were some NZ earthquakes he had to deal with)), If you do patterns-through-paths, you are going to recreate SELinux policy maintenance issues: maintaining the patterns globally is rather more abstract

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: add support for `else if` operator (#311)

2017-08-24 Thread Jeff Johnson
For example: Once can arrange for case-like set of inclusions by adding the value of %fedora to a filename and adding the snippets as SourceN: directives that are then selectively as a file path by an appropriate expansion. All I'm saying here is adding snippets as fedora_%{?fedora} files that

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: automatic creation of langpacks subpackages (#310)

2017-08-24 Thread Jeff Johnson
Creating package-per-language manually isn't exactly hard: see the macro at the link. (aside) For extra credit: someone ought to put up an example of how to map boilerplate macros into %SOURCEn directives that can be included (and %include is essentially just a %{cat %SOURCEn}) or inserted

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: automatic creation of langpacks subpackages (#310)

2017-08-24 Thread Jeff Johnson
I was referring to LINGUAS (which get text uses to determine what locales are included and installed). E.g. the Icelandic is.po file in RPM likely hasn't been touched for years: examining the contents shows that many strings have not been translated. LINGUAS can be used to disable installing

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: automatic creation of langpacks subpackages (#310)

2017-08-24 Thread Jeff Johnson
@Conan-Kudo: the "C" locale is the obvious choice for the "base" locale (which happens to also be English in most cases). -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Static code analysis task for make (#306)

2017-08-19 Thread Jeff Johnson
My previous comment was wrto adding static analysis instrumentation to rpmbuild for other packages. For static analysis on RPM itself by adding make targets, then LCOV/GCOV coverage targets and adding test cases is usually more useful than static analysis warnings. Note that I have years of

Re: [Rpm-maint] [rpm-software-management/rpm] Static code analysis task for make (#306)

2017-08-19 Thread Jeff Johnson
See https://scan.coverity.com/projects/rpm?tab=overview. Been there for years The make target I use looks like this: ``` # --- scan.coverity.com manual submission in $(PACKAGE).tgz COVDIR =cov-int COVBUILD = cov-build --dir $(COVDIR) COVDESC = $(shell date -u

Re: [Rpm-maint] [rpm-software-management/rpm] Static code analysis task for make (#306)

2017-08-19 Thread Jeff Johnson
History from the FreeBSD project showed that automating static analysis tools during build just leads to gigabytes of information that slowly composts because fixing warnings from static analysis is a Sisyphean task. The issue I am pointing out is that static analysis should NOT be run on every

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: extract changelog to file, include as %doc automatically and put stub in actual binary and source RPMs. (#305)

2017-08-19 Thread Jeff Johnson
@mattdm re "2. Shrink the size of /var/lib/rpm/Packages" There is simply no way to reduce the size of Packages in RPM without a drastic redesign because of "immutable header regions". Basically RPM metadata is signed/digested and delivered intact from build to install machine. Reducing the

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: extract changelog to file, include as %doc automatically and put stub in actual binary and source RPMs. (#305)

2017-08-19 Thread Jeff Johnson
That sure sounds like an issue for DNF, not RPM, then. Personally I think that there's a lot of deadweight in RPM packages, not only changelogs, but also including I18N in all its fantastic bloat, SELinux policies, and %config handling in its hoary rpmsave/rpmnew/rpmorig Baroque'ness. The

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-08-17 Thread Jeff Johnson
Meanwhile this patch adds a --queryformat modifier to generate UUID's from tag values, which is an entirely orthogonal and mostly non-intrusive usage case than converting hdrNum/tagNum to an octet string. -- You are receiving this because you are subscribed to this thread. Reply to this email

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-08-17 Thread Jeff Johnson
Changing indices to be octets (rather than integers) is a prerequisite to using UUID's as a join key. There are many problems (endianness, exposure of hdrNum/tagNum in the RPM API) that need to be solved to use a UUID as an octet string for LMDB (and for BDB). See other issues for

Re: [Rpm-maint] [rpm-software-management/rpm] rpm's doFoo function tends to expand twice (#313)

2017-09-01 Thread Jeff Johnson
Your comments would be easier to understand if you also included what your expectation actually is/was. Dumping a list of obscure expansions with no other clue, forces the reader to guess whatever you might consider the flaw. Just saying ... -- You are receiving this because you are

Re: [Rpm-maint] [rpm-software-management/rpm] rpm's doFoo function tends to expand twice (#313)

2017-09-01 Thread Jeff Johnson
Ah, got it. Lots depends on what you expect of %dirname as a "primitive". If you consider %dirname as, say, strrrchr(str, '/') with appropriate checks on return value, then there isn;t a needs for an expansion per se. But if you consider %dirname as a dirname(3) argument, then a fully expanded

Re: [Rpm-maint] [rpm-software-management/rpm] rpm's doFoo function tends to expand twice (#313)

2017-09-01 Thread Jeff Johnson
Arguably, removing the extra expansion for all of the above expansions returning strings might be less surprising (after thought): they are all string manipulations without side effects, and so the extra expansion isn't needed, and passing along '%%' is likelier more useful (at least with

Re: [Rpm-maint] [rpm-software-management/rpm] rpm's doFoo function tends to expand twice (#313)

2017-08-31 Thread Jeff Johnson
Um, yes, many of the doFoo() routines re-expand the substituted text. How exactly do you expect recursive substitution to Just Work if that was not happening?!? -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] multiple signatures support? (#189)

2017-10-18 Thread Jeff Johnson
@jcpunk ECM == Electronic Content Management signatures? Integrating RPM (actually GPG) signatures with ECM is a much more complex issue than whether multipole signatures are permitted into *.rpm package files. Treating a *.rpm file as static content with an appended ECM signature is entirely

Re: [Rpm-maint] [rpm-software-management/rpm] multiple signatures support? (#189)

2017-10-18 Thread Jeff Johnson
Ok. Elliptic curves are not supported by rpm4 (but are supported by rpm5). Meanwhile "multiple signatures" isn't the right issue to try to switch between RSA <-> EC signatures. Try an issue asking for "elliptic curve support" instead. -- You are receiving this because you are subscribed to

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-11-28 Thread Jeff Johnson
The signature padding tag cannot be part of an immutable region because its size/value can/will change. There is also little reason to include padding within plaintext used for digests/signatures As for ordering: you chose a different value for the padding tag than what is implemented in rpm5.

Re: [Rpm-maint] [rpm-software-management/rpm] Fail to check signature (#270)

2017-11-28 Thread Jeff Johnson
@pmatlai: you are claiming "might well be a bug" in rpm5 at bz#1514190 without supplying details. Either supply sufficient details (or a patch preferred) or stop claiming FUD, entirely your call. -- You are receiving this because you are subscribed to this thread. Reply to this email directly

Re: [Rpm-maint] [rpm-software-management/rpm] Fix error message for rpmlock (#355), by removing dead code (#358)

2017-11-24 Thread Jeff Johnson
I should point out that this is not "dead code" since the existence of a fcntl(2) read lock will prevent a write lock even if rpm itself does not expose the API to test read locks. That being said: your change is entirely sane, the fcntl(2) locking in RPM is badly flawed, made worse by

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Configurable macro file search path(s) (#363)

2017-11-24 Thread Jeff Johnson
Um, this simply isn't true. RPM has always searched multiple directories for macro definition files. See also the --macros CLI option (which takes a colon separated set of paths to load) to override the default value for macro search paths. -- You are receiving this because you are subscribed

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: Finer-grained symbol dependencies (#362)

2017-11-24 Thread Jeff Johnson
@miscroe: describing set versions as a "probabilistic method" with it a "small chance" of failure does not jibe with my understanding of the implementation. Please supply reasoning and a reproducer. AFAIK set versions is a clever (and faithful, no probabilistic operations) encoding of a set of

Re: [Rpm-maint] [rpm-software-management/rpm] Axe yet more unimplemented/unused/redundant cruft from query/verify flags (393879d)

2017-11-24 Thread Jeff Johnson
QUERY and VERIFY bits were consolidated int a single enum to *permit* patches to be applied from header content at some point, not otherwise. By all means, one can certainly chose to have separate QUERY and VEIFY logic bits. An equivalent amount (imho) of work on the implementation woul permit

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: make possible to pass multiple dirs to `_*dir` macro (#352)

2017-11-24 Thread Jeff Johnson
(aside) There are ways to configure rpmbuild to do everything in the current directory. I have published my macros configure file multiple times in the past 15 years. Adding multiple directories and a search path to existing macros like _sourcedir adds complexity for not much gain, and is quite

Re: [Rpm-maint] [rpm-software-management/rpm] RFE: make possible to pass multiple dirs to `_*dir` macro (#352)

2017-11-24 Thread Jeff Johnson
(aside) There are ways to configure rpmbuild to do everything in the current directory. I have published my macros configure file multiple times in the past 15 years. Adding multiple directories and a search path to existing macros like _sourcedir adds complexity for not much gain, and is quite

Re: [Rpm-maint] [rpm-software-management/rpm] Add an ArchiveX header (#372)

2017-12-13 Thread Jeff Johnson
The URL in a SourceN: tag is informative only, specifying where the tarball came from. Meanwhile, only the basename(2) of that URL has ever been used or needed by rpmbuild. The basename is appended to %_sourcedir where the tarball (or other source) is expected. There is no reason whatsoever

Re: [Rpm-maint] [rpm-software-management/rpm] Make %setup defaults smarter (#371)

2017-12-13 Thread Jeff Johnson
@nim-nim: please provide an explicit example of "named the same as the archive". Surely you do not mean that literally, where the directory name has, say, the suffix ".tar.gz" or ".zip" appended in the directory name. -- You are receiving this because you are subscribed to this thread. Reply

Re: [Rpm-maint] [rpm-software-management/rpm] Make %setup defaults smarter (#371)

2017-12-13 Thread Jeff Johnson
Also note that SourceN: is used for content that may not be an archive at all. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] rpm without TTY gives wrong error message (#355)

2017-11-11 Thread Jeff Johnson
Run as root. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/355#issuecomment-343713579___ Rpm-maint mailing list

Re: [Rpm-maint] [rpm-software-management/rpm] rpm without TTY gives wrong error message (#355)

2017-11-12 Thread Jeff Johnson
I assumed you wished rpm --erase to function. The important part of the error message is correct even if errno is incorrect when stdin is redirected. FWIW, I can't think of any rpm mode that actually reads from stdin (passwords are read using getpass(3)). Meanwhile errno is (likely) being

Re: [Rpm-maint] [rpm-software-management/rpm] scripts: add brp-mangle-shebangs (#344)

2017-11-03 Thread Jeff Johnson
Fussing with env-shebangs is the tip of a deeper problem: scripts can/will execute other executables, and there are no attempts to auto-generate rpm dependencies for other needed executables within scripts, including searching run-time (not build-time) PATH to see if executable exists. -- You

<    1   2   3   4   5   6   7   >