Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Panu Matilainen]

Closed #258.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#event-1789678681___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Panu Matilainen]

#187 got sorted out in the end, in a much simpler manner that doesn't pull in 
new dependencies. We can revisit the issue if an actual need for an event loop 
arises, but closing for now. Thanks for the effort though.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-413148321___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

@cgwalters when moving 100's of megabytes through a finite resource, there will 
be a measurable effect on the cache. that's true for any large I/O operation 
including rpm-ostree: have you run mincore to see how many pages are left in 
cache after syncfs? typically syncfs/sync guarantee write to disk, not removal 
from cache.

Meanwhile there are not non-blocking variants of fsync/fdatasync (which is 
essentially what an event loop provides, threads that can block, there are lots 
of ways to block without stopping an application), nor are there notification 
interfaces for any of these system calls.

Eliminating buffer bloat or adding priorities may improve performance, but will 
not solve the core issues, that filesystem durability/determinism increases 
reliability, and that 100's of Mb of I/O has an effect on system performance,.

The percentage of files that are generated by scripts in caches is vanishingly 
small when compared to static content imho. Wanna bet? ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320761934___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Colin Walters]

> all file operations are durable (in the sense that all installed files are 
> known written to disk).

Right now many real world systems rely on files written by scripts (for 
example, the bootloader configuration, `/etc/ld.so.cache`, etc.); librpm as 
designed today doesn't have visibility into those.  This is a problem that 
rpm-ostree fixes, because we generate a new root, and commit the entire thing 
(including scripts) into ostree, which itself handles `syncfs()` for objects.

> all dirty pages in the buffer cache are invalidated (so there is no I/O cache 
> blowout from performing an RPM upgrade).

I still think a chunk of this problem is in the kernel 
 - but userspace should probably allow some 
configuration here too...there is a `RWF_HIPRI` flag to `pwritev2`, maybe we 
could add a `RWF_LOWPRI` flag that was the opposite.  There is of course also 
the [block IO 
controller](https://www.kernel.org/doc/Documentation/cgroup-v1/blkio-controller.txt)
 - today since `rpm-ostree` always operates as a daemon, an admin could 
conveniently place it into a different cgroup.  But one could of course do 
something equivalent with [systemd 
run](https://utcc.utoronto.ca/~cks/space/blog/linux/SystemdForMemoryLimiting) 
wrapping `yum` or whatever.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320752370___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

FWIW, here is the rather pleasant and complete documentation for libeio:
[http://pod.tst.eu/http://cvs.schmorp.de/libeio/eio.pod](url)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320544792___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The libeio event loop adds RPM_CHECK_LIB to handle --with-libeio here:

[https://github.com/rpm5/rpm/commit/133c60b4036fd97af89390aea6ae94034239d545](url)
(Presumably you will do your own pkg-config AutoFU to handle --with-libeio)

The patch in comments above has been mildly reworked to remove all debugging 
and start the event loop globally here:

[https://github.com/rpm5/rpm/commit/f29874c11d30309048bc726f637c1cc5e2c6b856](url)

You will need to manually enable asynchronous fsync-on-close (similar as 
before):
```
diff --git a/lib/fsm.c b/lib/fsm.c
index 553774b..a7affcf 100644
--- a/lib/fsm.c
+++ b/lib/fsm.c
@@ -218,7 +218,7 @@ static int expandRegular(rpmfi fi, const char *dest, rpmpsm 
psm, int nodigest, i
 /* Create the file with 0200 permissions (write by owner). */
 {
mode_t old_umask = umask(0577);
-   wfd = Fopen(dest, "w.ufdio");
+   wfd = Fopen(dest, "wNS.ufdio");
umask(old_umask);
 }
 if (Ferror(wfd)) {
```

What is likely controversial is starting the event loop everywhere all the time 
in lib/poptALL.c
```
+#if defined(WITH_LIBEIO)
 +/* Start libeio event loop. */
 +rc = rpmeioStart(NULL);
 +if (rc)
 +  fprintf(stderr, _("%s: event loop failed to start (%d)\n"),
 +  xgetprogname(), rc);
 +#endif
```

If you don't like my patch in lib/poptALL.c (I have mixed feelings), then 
please supply a positive suggestion for where and how to start an event loop in 
RPM.

I doubt an idle event loop hurts anything. But if an idle event loop does 
affect RPM somehow, well its time to discover what else is needed IMHO. YMMV ...

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320544133___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The generic boilerplate part of this patch (see the rpm+fsync.gz patch in 
comment #2) is now committed at 
[https://github.com/rpm5/rpm/commit/d0d04ae67410ec1113b1ac8dab1c78a1fc2c003f](url)

You will need to enable/disable manually with this change in lib/fsm.c
```
diff --git a/lib/fsm.c b/lib/fsm.c
index 553774b..09f0354 100644
--- a/lib/fsm.c
+++ b/lib/fsm.c
@@ -218,7 +218,7 @@ static int expandRegular(rpmfi fi, const char *dest, rpmpsm 
psm, int nodigest, i
 /* Create the file with 0200 permissions (write by owner). */
 {
mode_t old_umask = umask(0577);
-   wfd = Fopen(dest, "w.ufdio");
+   wfd = Fopen(dest, "wS.ufdio");
umask(old_umask);
 }
 if (Ferror(wfd)) {
```

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320533949___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

@Conan-Kudo I do have something to add to RPM... filesystem durability and 
preventing cache blowout at modest cost. Not everything is a "patch".

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320525389___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

@n3npq I think he thought you had something ready to merge into RPM already. :)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320525269___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

@ignatenkobrain: send a PR ... for what?

This work is an improvement on PR #197 which hasn't moved in 6+ months.

And -- as stated -- the patches here are "rude-and-crude" and necessary to do 
the justification for adding an event loop to RPM. There is nothing here that 
should be added to RPM ... yet.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320525060___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

And one more try at speeding up fsync-on-close by adding more threads that can 
block:
```
eio_set_min_parallel(512);
eio_set_max_idle(512);
```

With 512 (the default is 4) backend threads that can block, the kernel install 
time is
```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.77user 1.10system 0:17.79elapsed 38%CPU (0avgtext+0avgdata 37968maxresident)k
0inputs+300120outputs (0major+21579minor)pagefuls 0swaps
```

And both fdatasync+fsync are being run (by inspection).

So the reasoning in the previous comment (~2.5x slower for fsync-on-close, 
quite acceptable, etc) seems correct correct even if I had a brain fart while 
tinkering ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320499489___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

Some minor tinkering on the fdatasync -> fsync pipeline with code like this:
```
pthread_yield();
if (eio_npending()) rc = eio_poll();
```
to minimize the number of calls to eio_poll() (which always processes at least 
one request) gives these timings

```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.67user 0.74system 0:17.65elapsed 36%CPU (0avgtext+0avgdata 29880maxresident)k
0inputs+300120outputs (0major+19150minor)pagefuls 0swaps
```

So using an event loop for asynchronous fsync-on-close is ~2.5x slower than not 
doing fsync, and has the added benefit)s) of filesystem durability and no I/O 
cache blowout on large upgrades.

That is a quite acceptable overhead for always doing fsync-on-close IMHO: YMMV 
as always.

Note that no scripts are running in my silly 1 kernel upgrade testing: 
overlapping fsync-on-close with scriptlet execution may result in further 
improvements in The Real World.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320493837___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

There is one other alternative: libcoro (also by Marc Lehman). If the only 
problem I was trying to solve was making fdatasync+fsync asynchronous, then 
libcoro would be the best choice. My patch essentially reduces libeio to 
co-routine functionality. libcoro does not require a scheduler, nor threads, 
etc if there is a non-blocking variant of an existing system call that needs to 
be run.

Meanwhile there are serious design issues switching to a coroutine 
implementation, the code is often quite mysterious.

There are similar issues reworking the design of existing imperative I/O loops 
to run asynchronously. However, the issues that need to be dealt with in an 
event loop are rather easier than redesigning with coroutines.

Again, JMHO, YMMV, everyone's does. libeio is what I've chosen to attempt an 
event loop implementation in RPM, try-and-see-and-measure is the only 
development paradigm I believe in ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320491272___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Igor Gnatenko]

@n3npq all this is great, but you have to send Pull Request.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320490451___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The only sane way to add an event loop within RPM is through embedding, at 
which point maintenance is with RPM, not through actively maintained external 
libraries. libeio is the only reasonable embeddable choice.

Extending external libeio with additional system calls is trickier with 
external libeio as well because libeio includes both high level wrappings as 
well as a low level core engine. Again embedding is easier to extend and 
straightforward to maintain (if extending libeio to additional system calls 
like posix_fadvise, or David Howell's new-fangled kernel xstat system call). 
Meanwhile what is on libeio suffices for most usage cases, now and for the 
future, imho.

Do not fool yourself thinking otherwise: meanwhile libelo is mature (i.e slowly 
changing) and well tested, and Marc Lehman responds quickly to questions (also 
tested by me ;-)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320490349___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

Thanks for the reasoning. 👍 

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489857___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The main reason why libevent was not used:

* Marc Lehmann  chose to move on from libevent -> 
libev+libeio and strip out complex buffer handling. Small and targeted and 
embeddable are all KISS; automating buffer allocation handling is hardly useful 
or necessary to rpmio.

The main reason why libuv (and nodejs) are not appropriate for RPM:
* libuv/nodejs are about asynchronous networking: nothing much within RPM4 I/O 
needs/uses any network related system calls, so most of libuv is overkill.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489822___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

libeio is fine, I just wanted to know if you had considered the alternatives 
and why this one was picked. It looks like libeio is still actively maintained, 
which is good.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489065___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

Yes, there is libevent, then libev+libeio, and now libuv, all tuned towards 
slightly different goals.

libeio is narrowly targeted at asynchronous system calls, can be embedded, and 
is generally lean-and-mean and well tested.

But go shopping for alternatives if you wish: I chose libeio after looking at 
multiple alternatives.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320489014___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[ニール・ゴンパ]

@n3npq I thought nodejs used [libuv](http://libuv.org/) nowadays? AFAIK, they 
originally used [libeio](http://software.schmorp.de/pkg/libeio.html) + 
[libev](http://software.schmorp.de/pkg/libev.html), but created libuv later on.

Do you have any particular insight into the differences (pluses/minuses) of 
using libuv vs libeio+libev?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320485287___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The attached patch does a rude-and-crude libeio pipe driven event loop to ramie.

The patch adds an 'N' (mnemonic: O_NONBLOCK) flag to Fopen() flags.

The libeio loop is lazily started, and atexit(3) is used to wait for all events 
to be completed.

The operations of fdatasync/fadvise/fsync are pipelined (i.e. fdatasync/fsync 
are each separate events).

Basically libeio is being used as a co-routine monitor to move the costly 
operations off  the main thread of operation in order to do a wall clock 
benchmark:

```
$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm
...
5.67user 0.88system 0:39.75elapsed 16%CPU (0avgtext+0avgdata 32420maxresident)k
0inputs+22outputs (0major+13681minor)pagefuls 0swaps

```

So ~5x slower than no fsync(3) which is currently implemented in RPM.

Which is starting to get to a reasonable execution cost for the simultaneous 
benefits of
* all file operations are durable (in the sense that all installed files are 
known written to disk).
* all dirty pages in the buffer cache are invalidated (so there is no I/O cache 
blowout from performing an RPM upgrade).

The underlying goal here was to assess whether a libeio event loop (which is 
far more general than fdatasync/fsync operations here) has benefits if 
implemented in RPM. The rude-and-crude implementation here is already 2x faster 
than the synchronous rsync-on-close implementation, and is not much more 
complex. JMHO, YMMV, everyone's does.

[rpm+fsync+libeio.patch.gz](https://github.com/rpm-software-management/rpm/files/1202849/rpm.fsync.libeio.patch.gz)

The next step(s) will be to do a proper RPM+LIBEIO implementation so that 
libeio can be used everywhere as needed, not just on the peculiar 
fire-and-forget fsync-on-close file write path used here.


-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320484006___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Adding an event loop in RPM to run asynchronous operations. (#258)

[Jeff Johnson]

The attached patch adds allocate/fadvise/fdatasync/fsync functions in rpmio 
with the calls on I/O intensive install code paths.

In addition, two flags have been added to Fopen(): 'S' to do fsync-on-close, 
and '?' to do per-fd debugging.

Simple benchmarks for a kernel package installation (rechecking PR #197 
measurements)

without fsync-on-close:

$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm 
...
5.67user 0.54system 0:07.45elapsed 83%CPU (0avgtext+0avgdata 29780maxresident)k
0inputs+303336outputs (14major+18667minor)pagefuls 0swaps

with fsync-on-close

$ sudo /usr/bin/time ./rpm -U --root=/var/tmp/xxx --nodeps --force 
kernel-3.10.0-514.26.2.el7.x86_64.rpm 
...
5.70user 0.87system 1:15.05elapsed 8%CPU (0avgtext+0avgdata 29772maxresident)k
8inputs+306472outputs (14major+19052minor)pagefuls 0swaps

So fsync-on-close is  ~10x slower on whatever disk I happen to be using on 
CentOS7.

The next step will be to use libeio to run fdatasync+fadvise+fsync+close 
asynchronously.

[rpm+fsync.patch.gz](https://github.com/rpm-software-management/rpm/files/1199322/rpm.fsync.patch.gz)




-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/258#issuecomment-320161172___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint