Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

FWIW, here is the rather pleasant and complete documentation for libeio:
[http://pod.tst.eu/http://cvs.schmorp.de/libeio/eio.pod](url)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320544792___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The libeio event loop adds RPM_CHECK_LIB to handle --with-libeio here:

[https://github.com/rpm5/rpm/commit/133c60b4036fd97af89390aea6ae94034239d545](url)
(Presumably you will do your own pkg-config AutoFU to handle --with-libeio)

The patch in comments above has been mildly reworked to remove all debugging 
and start the event loop globally here:

[https://github.com/rpm5/rpm/commit/f29874c11d30309048bc726f637c1cc5e2c6b856](url)

You will need to manually enable asynchronous fsync-on-close (similar as 
before):
```
diff --git a/lib/fsm.c b/lib/fsm.c
index 553774b..a7affcf 100644
--- a/lib/fsm.c
+++ b/lib/fsm.c
@@ -218,7 +218,7 @@ static int expandRegular(rpmfi fi, const char *dest, rpmpsm 
psm, int nodigest, i
 /* Create the file with 0200 permissions (write by owner). */
 {
mode_t old_umask = umask(0577);
-   wfd = Fopen(dest, "w.ufdio");
+   wfd = Fopen(dest, "wNS.ufdio");
umask(old_umask);
 }
 if (Ferror(wfd)) {
```

What is likely controversial is starting the event loop everywhere all the time 
in lib/poptALL.c
```
+#if defined(WITH_LIBEIO)
 +/* Start libeio event loop. */
 +rc = rpmeioStart(NULL);
 +if (rc)
 +  fprintf(stderr, _("%s: event loop failed to start (%d)\n"),
 +  xgetprogname(), rc);
 +#endif
```

If you don't like my patch in lib/poptALL.c (I have mixed feelings), then 
please supply a positive suggestion for where and how to start an event loop in 
RPM.

I doubt an idle event loop hurts anything. But if an idle event loop does 
affect RPM somehow, well its time to discover what else is needed IMHO. YMMV ...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320544133___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding ZSTD (or other FSE compressor) (#256)

[Jeff Johnson]

This patch is now committed at 
https://github.com/rpm5/rpm/commit/debde46a23c5a942d017026903020eb3b389d91c

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/256#issuecomment-320532669___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an LMDB backend to RPM (#281)

[Jeff Johnson]

The final patch (with minor changes) is is now committed at 
https://github.com/rpm5/rpm/commit/528925bd9967d9b741787df56e46a5c83c7a54aa

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/281#issuecomment-320532173___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding header tag formats to generate UUIDS (#269)

[Jeff Johnson]

The rpm_uuid.patch is now applied to an RPM tree at 
[https://github.com/rpm5/rpm/commit/c8c72fb195790dfa2d40c20ca597d9cc2b75](url)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/269#issuecomment-320530188___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Find lang.sh multi names (#235)

[proyvind]

proyvind commented on this pull request.

This should be sufficient for approval, no? :)



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/235#pullrequestreview-54542322___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

@Conan-Kudo I do have something to add to RPM... filesystem durability and 
preventing cache blowout at modest cost. Not everything is a "patch".

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320525389___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

@n3npq I think he thought you had something ready to merge into RPM already. :)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320525269___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

@ignatenkobrain: send a PR ... for what?

This work is an improvement on PR #197 which hasn't moved in 6+ months.

And -- as stated -- the patches here are "rude-and-crude" and necessary to do 
the justification for adding an event loop to RPM. There is nothing here that 
should be added to RPM ... yet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320525060___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

And one more try at speeding up fsync-on-close by adding more threads that can 
block:
```
eio_set_min_parallel(512);
eio_set_max_idle(512);
```

With 512 (the default is 4) backend threads that can block, the kernel install 
time is
```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.77user 1.10system 0:17.79elapsed 38%CPU (0avgtext+0avgdata 37968maxresident)k
0inputs+300120outputs (0major+21579minor)pagefuls 0swaps
```

And both fdatasync+fsync are being run (by inspection).

So the reasoning in the previous comment (~2.5x slower for fsync-on-close, 
quite acceptable, etc) seems correct correct even if I had a brain fart while 
tinkering ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320499489___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

Some minor tinkering on the fdatasync -> fsync pipeline with code like this:
```
pthread_yield();
if (eio_npending()) rc = eio_poll();
```
to minimize the number of calls to eio_poll() (which always processes at least 
one request) gives these timings

```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.67user 0.74system 0:17.65elapsed 36%CPU (0avgtext+0avgdata 29880maxresident)k
0inputs+300120outputs (0major+19150minor)pagefuls 0swaps
```

So using an event loop for asynchronous fsync-on-close is ~2.5x slower than not 
doing fsync, and has the added benefit)s) of filesystem durability and no I/O 
cache blowout on large upgrades.

That is a quite acceptable overhead for always doing fsync-on-close IMHO: YMMV 
as always.

Note that no scripts are running in my silly 1 kernel upgrade testing: 
overlapping fsync-on-close with scriptlet execution may result in further 
improvements in The Real World.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320493837___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

There is one other alternative: libcoro (also by Marc Lehman). If the only 
problem I was trying to solve was making fdatasync+fsync asynchronous, then 
libcoro would be the best choice. My patch essentially reduces libeio to 
co-routine functionality. libcoro does not require a scheduler, nor threads, 
etc if there is a non-blocking variant of an existing system call that needs to 
be run.

Meanwhile there are serious design issues switching to a coroutine 
implementation, the code is often quite mysterious.

There are similar issues reworking the design of existing imperative I/O loops 
to run asynchronously. However, the issues that need to be dealt with in an 
event loop are rather easier than redesigning with coroutines.

Again, JMHO, YMMV, everyone's does. libeio is what I've chosen to attempt an 
event loop implementation in RPM, try-and-see-and-measure is the only 
development paradigm I believe in ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320491272___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The only sane way to add an event loop within RPM is through embedding, at 
which point maintenance is with RPM, not through actively maintained external 
libraries. libeio is the only reasonable embeddable choice.

Extending external libeio with additional system calls is trickier with 
external libeio as well because libeio includes both high level wrappings as 
well as a low level core engine. Again embedding is easier to extend and 
straightforward to maintain (if extending libeio to additional system calls 
like posix_fadvise, or David Howell's new-fangled kernel xstat system call). 
Meanwhile what is on libeio suffices for most usage cases, now and for the 
future, imho.

Do not fool yourself thinking otherwise: meanwhile libelo is mature (i.e slowly 
changing) and well tested, and Marc Lehman responds quickly to questions (also 
tested by me ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320490349___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

Thanks for the reasoning.  

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489857___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The main reason why libevent was not used:

* Marc Lehmann  chose to move on from libevent -> 
libev+libeio and strip out complex buffer handling. Small and targeted and 
embeddable are all KISS; automating buffer allocation handling is hardly useful 
or necessary to rpmio.

The main reason why libuv (and nodejs) are not appropriate for RPM:
* libuv/nodejs are about asynchronous networking: nothing much within RPM4 I/O 
needs/uses any network related system calls, so most of libuv is overkill.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489822___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

libeio is fine, I just wanted to know if you had considered the alternatives 
and why this one was picked. It looks like libeio is still actively maintained, 
which is good.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489065___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

Yes, there is libevent, then libev+libeio, and now libuv, all tuned towards 
slightly different goals.

libeio is narrowly targeted at asynchronous system calls, can be embedded, and 
is generally lean-and-mean and well tested.

But go shopping for alternatives if you wish: I chose libeio after looking at 
multiple alternatives.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489014___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint