Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
There is one other alternative: libcoro (also by Marc Lehman). If the only problem I was trying to solve was making fdatasync+fsync asynchronous, then libcoro would be the best choice. My patch essentially reduces libeio to co-routine functionality. libcoro does not require a scheduler, nor

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread ニール・ゴンパ
libeio is fine, I just wanted to know if you had considered the alternatives and why this one was picked. It looks like libeio is still actively maintained, which is good. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
Yes, there is libevent, then libev+libeio, and now libuv, all tuned towards slightly different goals. libeio is narrowly targeted at asynchronous system calls, can be embedded, and is generally lean-and-mean and well tested. But go shopping for alternatives if you wish: I chose libeio after

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Igor Gnatenko
@n3npq all this is great, but you have to send Pull Request. -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread ニール・ゴンパ
Thanks for the reasoning.  -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489857___ Rpm-maint mailing

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
The main reason why libevent was not used: * Marc Lehmann chose to move on from libevent -> libev+libeio and strip out complex buffer handling. Small and targeted and embeddable are all KISS; automating buffer allocation handling is hardly useful or necessary to rpmio. The

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
The only sane way to add an event loop within RPM is through embedding, at which point maintenance is with RPM, not through actively maintained external libraries. libeio is the only reasonable embeddable choice. Extending external libeio with additional system calls is trickier with external

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
Some minor tinkering on the fdatasync -> fsync pipeline with code like this: ``` pthread_yield(); if (eio_npending()) rc = eio_poll(); ``` to minimize the number of calls to eio_poll() (which always processes at least one request) gives these timings ``` $ sudo /usr/bin/time

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
And one more try at speeding up fsync-on-close by adding more threads that can block: ``` eio_set_min_parallel(512); eio_set_max_idle(512); ``` With 512 (the default is 4) backend threads that can block, the kernel install time is ``` $ sudo /usr/bin/time ./rpm -U

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread ニール・ゴンパ
@n3npq I think he thought you had something ready to merge into RPM already. :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

2017-08-06 Thread Jeff Johnson
The rpm_uuid.patch is now applied to an RPM tree at [https://github.com/rpm5/rpm/commit/c8c72fb195790dfa2d40c20ca597d9cc2b75](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
@Conan-Kudo I do have something to add to RPM... filesystem durability and preventing cache blowout at modest cost. Not everything is a "patch". -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Find lang.sh multi names (#235)

2017-08-06 Thread proyvind
proyvind commented on this pull request. This should be sufficient for approval, no? :) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
@ignatenkobrain: send a PR ... for what? This work is an improvement on PR #197 which hasn't moved in 6+ months. And -- as stated -- the patches here are "rude-and-crude" and necessary to do the justification for adding an event loop to RPM. There is nothing here that should be added to RPM

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

2017-08-06 Thread Jeff Johnson
The final patch (with minor changes) is is now committed at https://github.com/rpm5/rpm/commit/528925bd9967d9b741787df56e46a5c83c7a54aa -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
The libeio event loop adds RPM_CHECK_LIB to handle --with-libeio here: [https://github.com/rpm5/rpm/commit/133c60b4036fd97af89390aea6ae94034239d545](url) (Presumably you will do your own pkg-config AutoFU to handle --with-libeio) The patch in comments above has been mildly reworked to remove

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

2017-08-06 Thread Jeff Johnson
This patch is now committed at https://github.com/rpm5/rpm/commit/debde46a23c5a942d017026903020eb3b389d91c -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub:

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
The generic boilerplate part of this patch (see the rpm+fsync.gz patch in comment #2) is now committed at [https://github.com/rpm5/rpm/commit/d0d04ae67410ec1113b1ac8dab1c78a1fc2c003f](url) You will need to enable/disable manually with this change in lib/fsm.c ``` diff --git a/lib/fsm.c

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

2017-08-06 Thread Jeff Johnson
FWIW, here is the rather pleasant and complete documentation for libeio: [http://pod.tst.eu/http://cvs.schmorp.de/libeio/eio.pod](url) -- You are receiving this because you are subscribed to this thread. Reply to this email directly or view it on GitHub: