Re: [Rpm-maint] [rpm-software-management/rpm] Increase lmdb DB size from 256M to 1G (#902)

[Panu Matilainen]

Merged #902 into master.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/902#event-2720110416___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Increase lmdb DB size from 256M to 1G (#902)

[Panu Matilainen]

pmatilai approved this pull request.

Yeah, the need to predetermine the database size is one of the shortcomings of 
LMDB. On 64bit platforms that support sparse files, we could just some huge 
size and forget, but even 1GB is an enormous chunk of the available address 
space on 32bit.

Anyway, 256MB is really on the small side in any case, so until a better 
solution comes along...



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/902#pullrequestreview-303036942___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fixes for some issues on Arm platforms (#901)

[Panu Matilainen]

pmatilai requested changes on this pull request.

As noted above, this needs to be worked out between various Arm maintainers to 
come up with solutions that work for everybody. We're not on a seesaw.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/901#pullrequestreview-303016842___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fixes for some issues on Arm platforms (#901)

[Panu Matilainen]

Have I ever mentioned I hate arm patches, because the only guaranteed thing is 
that nobody agrees on how they should be, and I can't be the judge since I 
don't have a clue.

The commits being reversed have gone in to fix somebody elses issues, and we 
can't revert them just like that. You'll need to work with the other Arm people 
to come up with solutions that work for everybody.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/901#issuecomment-543014765___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[Panu Matilainen]

pmatilai commented on this pull request.



> @@ -59,6 +64,15 @@ dbDetectBackend(rpmdb rdb)
 }
 #endif
 
+#if defined(WITH_SQLITE)
+path = rstrscat(NULL, dbhome, "/Packages.sqlite", NULL);

The thought has crossed my mind as well. The "reason" to have it resemble the 
others is basically to allow covering the different variants with "Packages*" 
but then LMDB doesn't follow that and we'll need an API to cover for the 
differences anyway so yeah, we could just as well call it something more 
sensible.

In general, naming (including database columns) is pretty much in the open 
still.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899#discussion_r335816977___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[Panu Matilainen]

@mlschroe , choice is good, right? ;)

There's value in having a minimal built-in solution, and there's value in 
having a solution based on proven standard building blocks, they just appeal to 
different audiences. And right now we don't really have the latter: BDB is a 
six years unmaintained behemoth nobody wants to touch with a ten foot pole, 
LMDB is three years waiting for upstream to up the key size, and we're stuck in 
the middle. This is intended to fill that particular gap.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899#issuecomment-543010249___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fixes for some issues on Arm platforms (#901)

[ニール・ゴンパ]

Note that even that fix as it stands will break OpenMandriva, who wrote that to 
support their distribution. If you want to alter it, work with @berolinux to 
fix it properly.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/901#issuecomment-542952741___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fixes for some issues on Arm platforms (#901)

[ニール・ゴンパ]

@nullr0ute The only potentially legitimate commit here is 
https://github.com/rpm-software-management/rpm/pull/901/commits/790b0ec08027ba1676acc4091045d78e65aa8f2d,
 as this one is actually about fixing the problem referenced in the RH bug 
linked in the PR description.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/901#issuecomment-542951683___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Fixes for some issues on Arm platforms (#901)

[ニール・ゴンパ]

Conan-Kudo requested changes on this pull request.

This whole PR is awful. 

Aside from the fact that all but one of the commits reverted are mine, most of 
those changes were made as part of upstreaming ARM enablement from Mageia. The 
last commit was upstreaming changes from OpenMandriva to support the ARM 
architectures they support.

And the commit about `aarch64` and `arm64` "not being equivalent": tough. In 
virtually *every* single non-rpm/non-gcc platform, they *are* equivalent. We 
did the same thing for `amd64` and `ia32e` for `x86_64`.



-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/901#pullrequestreview-302955624___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[ニール・ゴンパ]

@mlschroe @pmatilai Well, @midipix and @Redfoxmoon3 are experimenting with the 
ndb backend for rpm-based [midipix](https://midipix.org/). It's been 
interesting...

I'm more interested in the SQLite backend simply because the tooling is better 
for diagnosing and repairing SQLite databases.

(cc: @dmach)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899#issuecomment-542949706___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[ニール・ゴンパ]

Conan-Kudo requested changes on this pull request.



> @@ -59,6 +64,15 @@ dbDetectBackend(rpmdb rdb)
 }
 #endif
 
+#if defined(WITH_SQLITE)
+path = rstrscat(NULL, dbhome, "/Packages.sqlite", NULL);

For this, we only have the single database file, right? Couldn't we just call 
this `rpmdb.sqlite` instead of `Packages.sqlite`?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899#pullrequestreview-302953172___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Increase lmdb DB size from 256M to 1G (#902)

[ifel]

The mapsize in LMDB is a limit of the DB size. It's set to 256M now, and if 
the DB reaches this size, rpm operations fails with:
---
error: lmdb:rc(-30792) = mdb_cursor_put(): MDB_MAP_FULL: Environment 
mapsize limit reached
error: rc(-30792) adding header #31959 record: MDB_MAP_FULL: Environment 
mapsize limit reached
---

In my case, it was next to it, and it was failing on updating an RPM with 34.5K 
of files (as during update it should store data for the old and new files both).

Increase the size of the DB to 1GB, it should cover most of use cases now, but 
we should probably come up with a better solution in the future, if we decide 
to use this DB.
You can view, comment on, or merge this pull request online at:

  https://github.com/rpm-software-management/rpm/pull/902

-- Commit Summary --

  * Increase lmdb DB size from 256M to 1G

-- File Changes --

M lib/backend/lmdb.c (2)

-- Patch Links --

https://github.com/rpm-software-management/rpm/pull/902.patch
https://github.com/rpm-software-management/rpm/pull/902.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/902
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Fixes for some issues on Arm platforms (#901)

[Peter Robinson]

We're seeing a few issues on Arm platforms [1], ARMv7 and aarch64, and 
review there's a few issues and clarifications needed around the way some 
of the Arm features work. This fixes up the issues seen and clarifies a few 
details of some of the Arm architecture quirks.

[1] https://bugzilla.redhat.com/show_bug.cgi?id=1562084

Signed-off-by: Peter Robinson 
You can view, comment on, or merge this pull request online at:

  https://github.com/rpm-software-management/rpm/pull/901

-- Commit Summary --

  * Revert "rpmrc: Add architecture compatibility mapping between aarch64 
and arm64"
  * Revert "Fix the armv5tl arch compatibility list"
  * Revert "Add armv5tl support"
  * Remove problematic sub variants of armv8 and related

-- File Changes --

M macros.in (2)
M rpmrc.in (30)

-- Patch Links --

https://github.com/rpm-software-management/rpm/pull/901.patch
https://github.com/rpm-software-management/rpm/pull/901.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/901
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] Rpm database backend benchmarks

[Panu Matilainen]

On 10/16/19 4:28 PM, Michael Schroeder wrote:


Hi,

I wrote a little benchmarking tool to find out how the different
database backends compare. The backends tested are bdb, ndb,
lmdb, and the new sqlite backend.

I used the 2506 packages I have on my system as data set. The
benchmark tool does the following:

Add all packages to an empty database, in forward/reverse/random
order.

Remove all packages from a database that contains all packages,
in forward/reverse/random order.

Update all packages in forward/reverse/random order. An update
consistes of an reinstall of the package plus a remove of the
old header

Simulate a dependency check of all packages (rpm -Va --nofiles).


Can you share this benchmark tool?

I ran similar comparisons (rpm -Uvh --justdb, rpmdb --rebuilddb, rpm -e 
--test all etc) manually when doing the initial work on the sqlite 
backend, sometimes with modifications to bdb/lmdb make the comparisons 
more relevant (enable transactions, syncs etc), and mostly just to see 
it's in the ballpark with others.




The install/erase/update lines look like this:

Operation
 forward / reverse / random in seconds with fsync disabled
 forward / reverse / random in seconds with fsync enabled
 forward / reverse / random disk space used in MByte

Here's the result:

BDB
---
Adding all headers...
 3.19s / 3.21s / 3.17s
 85.69s / 88.58s / 84.99s
 164.41M / 164.48M / 164.52M
Erasing all headers...
 3.74s / 3.70s / 3.70s
 70.00s / 70.06s / 74.60s
 164.29M / 164.30M / 164.29M
Updating all headers...
 7.23s / 7.31s / 7.26s
 147.16s / 150.98s / 147.26s
 174.17M / 174.07M / 174.17M
Dep check
 4.82s

NDB
---
Adding all headers...
 1.05s / 0.99s / 1.04s
 46.91s / 41.94s / 42.09s
 177.70M / 175.22M / 173.37M
Erasing all headers...
 1.10s / 0.95s / 1.09s
 42.47s / 30.93s / 36.13s
 0.67M / 0.71M / 0.82M
Updating all headers...
 2.34s / 2.42s / 2.61s
 78.17s / 70.24s / 72.18s
 164.97M / 170.09M / 168.97M
Dep check
 2.81s

LMDB

Adding all headers...
 1.06s / 1.03s / 1.03s
 (not implemented)
 268.44M / 268.44M / 268.44M
Erasing all headers...
 1.13s / 1.14s / 1.13s
 (not implemented)
 268.44M / 268.44M / 268.44M
Updating all headers...
 3.33s / 3.37s / 3.42s
 (not implemented)
 268.44M / 268.44M / 268.44M
Dep check
 2.36s

SQLITE
--
Adding all headers...
 4.24s / 4.28s / 4.24s
 34.50s / 38.03s / 34.48s
 158.58M / 158.54M / 158.58M
Erasing all headers...
 19.58s / 19.59s / 20.52s
 51.12s / 55.65s / 51.83s
 158.58M / 158.58M / 158.58M
Updating all headers...
 45.52s / 45.84s / 46.39s
 108.55s / 114.18s / 113.97s
 172.46M / 171.74M / 173.19M
Dep check
 12.50s


Things to note:

- Berkeley db is actually not that fast, both lmdb and ndb are much
   faster


Yup. And it gets slower if you enable transactions.


- rpm's lmdb code does not implement fsync


Yup. Drop MDB_MAPASYNC and MDB_WRITEMAP flags to level the playground. 
It's still fastest of the lot, but much more comparable.



- there's something weird with the sqlite package erase, it takes
   way too much time with fsync disabled


Right, haven't tested erase with fsync disabled because to me it's a 
rare corner case (as opposed to install). With sqlite, if it goes too 
slow there are simply too many transactions going on (which is one thing 
that makes it quite different from bdb/lmdb). Sqlite doesn't really have 
a "disable fsync" mode in the sense that eg bdb has, it's more a matter 
of finding the right balance of transaction sizes and pragmas etc.


Probably not even related, but for one the current rpmdb backend API 
forces it to play on the key-value db terms, whereas internally it could 
just do one sweeping "delete from ..." statement and skip a whole lot of 
work currently done if only the API permitted that (working on it).



- sqlite is quite slow


It's also not an entirely apples-to-apples comparison, in many ways.

In my benchmarks, LMDB was the all-round winner even when syncing 
enabled and modified to use per-package transactions, but then it's a 
bit moot to compare as long as it can't be used for real due to the key 
size limitation.


Besides sqlite having to do horribly stupid things due to the key-value 
oriented internal backend API, the new backend is also not really 
optimized at all. So far I've only cared about getting it into the rough 
ballpark with BDB backend which is the thing it's supposed to be 
eventually replacing. For various things it's faster even in its current 
state, which is not a bad start. It's just a totally different animal 
from the others so it needs rather different approaches to make it fast(er).



- sqlite's "Adding all header" benchmark is quite fast with fsync
   enabled. I wonder what sqlite guarantees if there is a crash


Generally speaking, all sqlite operations are transactionally protected 
and thus crash-resilient (un

[Rpm-maint] Rpm database backend benchmarks

[Michael Schroeder]

Hi,

I wrote a little benchmarking tool to find out how the different
database backends compare. The backends tested are bdb, ndb,
lmdb, and the new sqlite backend.

I used the 2506 packages I have on my system as data set. The
benchmark tool does the following:

Add all packages to an empty database, in forward/reverse/random
order.

Remove all packages from a database that contains all packages,
in forward/reverse/random order.

Update all packages in forward/reverse/random order. An update
consistes of an reinstall of the package plus a remove of the
old header

Simulate a dependency check of all packages (rpm -Va --nofiles).

The install/erase/update lines look like this:

Operation
forward / reverse / random in seconds with fsync disabled
forward / reverse / random in seconds with fsync enabled
forward / reverse / random disk space used in MByte

Here's the result:

BDB
---
Adding all headers...
3.19s / 3.21s / 3.17s
85.69s / 88.58s / 84.99s
164.41M / 164.48M / 164.52M
Erasing all headers...
3.74s / 3.70s / 3.70s
70.00s / 70.06s / 74.60s
164.29M / 164.30M / 164.29M
Updating all headers...
7.23s / 7.31s / 7.26s
147.16s / 150.98s / 147.26s
174.17M / 174.07M / 174.17M
Dep check
4.82s

NDB
---
Adding all headers...
1.05s / 0.99s / 1.04s
46.91s / 41.94s / 42.09s
177.70M / 175.22M / 173.37M
Erasing all headers...
1.10s / 0.95s / 1.09s
42.47s / 30.93s / 36.13s
0.67M / 0.71M / 0.82M
Updating all headers...
2.34s / 2.42s / 2.61s
78.17s / 70.24s / 72.18s
164.97M / 170.09M / 168.97M
Dep check
2.81s

LMDB

Adding all headers...
1.06s / 1.03s / 1.03s
(not implemented)
268.44M / 268.44M / 268.44M
Erasing all headers...
1.13s / 1.14s / 1.13s
(not implemented)
268.44M / 268.44M / 268.44M
Updating all headers...
3.33s / 3.37s / 3.42s
(not implemented)
268.44M / 268.44M / 268.44M
Dep check
2.36s

SQLITE
--
Adding all headers...
4.24s / 4.28s / 4.24s
34.50s / 38.03s / 34.48s
158.58M / 158.54M / 158.58M
Erasing all headers...
19.58s / 19.59s / 20.52s
51.12s / 55.65s / 51.83s
158.58M / 158.58M / 158.58M
Updating all headers...
45.52s / 45.84s / 46.39s
108.55s / 114.18s / 113.97s
172.46M / 171.74M / 173.19M
Dep check
12.50s


Things to note:

- Berkeley db is actually not that fast, both lmdb and ndb are much
  faster
- rpm's lmdb code does not implement fsync
- there's something weird with the sqlite package erase, it takes
  way too much time with fsync disabled
- sqlite is quite slow
- sqlite's "Adding all header" benchmark is quite fast with fsync
  enabled. I wonder what sqlite guarantees if there is a crash
- ndb is the only database that can shrink if packages get
  removed
- lmdb won the dep check benchmark. I think this is because it
  mmaps the complete database and thus has not to copy the header
  data.

If fsync is enabled (aka normal rpm operation), all implementations
take very long. The question is how much this is drowned out by the
time spent in unpacking/erasing all the files on disk.

Cheers,
  Michael.

-- 
Michael Schroeder   m...@suse.de
SUSE LINUX GmbH,   GF Jeff Hawn, HRB 16746 AG Nuernberg
main(_){while(_=~getchar())putchar(~_-1/(~(_|32)/13*2-11)*13);}
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Fully shutdown DBUS on systemd_inhibit cleanup (RhBug:1714657)" (#900)

[Colin Walters]

Maybe key off `rpmsqSetInterruptSafety()`?  I don't think mock calls that yet 
but it probably should.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/900#issuecomment-542695270___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Fully shutdown DBUS on systemd_inhibit cleanup (RhBug:1714657)" (#900)

[Colin Walters]

Oh wow.  Is there a way for me to unconditionally turn this off in rpm-ostree?  
Since our upgrades are always offline+transactional we never need it.

More generally, it seems like this plugin should be a no-op if we're installing 
into a root != `/` for e.g. the mock case right?

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/900#issuecomment-542693704___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Fully shutdown DBUS on systemd_inhibit cleanup (RhBug:1714657)" (#900)

[Panu Matilainen]

Merged #900 into master.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/900#event-2717192640___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Revert "Fully shutdown DBUS on systemd_inhibit cleanup (RhBug:1714657)" (#900)

[ニール・ゴンパ]

Conan-Kudo approved this pull request.





-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/900#pullrequestreview-302470416___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add NEVR provides for all packages that would be built into source rpms (#891)

[Panu Matilainen]

Tagging #657 for are interesting, related ideas

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/891#issuecomment-542616720___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Make %{buildsubdir} usable without being a side effect of %setup. (#860)

[Panu Matilainen]

#551 is relevant/related.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/860#issuecomment-542613820___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] bad version comparison in all rpm (#896)

[Michael Schroeder]

(See also issue #561 )

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/issues/896#issuecomment-542611714___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Enable -Werror in CI builds (#898)

[Panu Matilainen]

Merged #898 into master.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/898#event-2716838543___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Enable -Werror in CI builds (#898)

[Panu Matilainen]

Since nobody objects...

Note that rawhide introducing a new gcc might introduce new warnings that then 
completely block our CI, if that happens we'll just disable -Werror on rawhide 
until dealt with.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/898#issuecomment-542607088___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Multiple ndb improvements (#895)

[Panu Matilainen]

Roger that.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/895#issuecomment-542605858___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Multiple ndb improvements (#895)

[Panu Matilainen]

Merged #895 into master.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/895#event-2716829046___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[Michael Schroeder]

I wouldn't dismiss poor ndb that easily. It has not many LoC and fits rpm's 
needs perfectly. The packages database is similar to what rpm-3 had in the past 
(but written with fault tolerance in mind), and the indexes are just mmaped 
hash tables.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899#issuecomment-542599750___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Multiple ndb improvements (#895)

[Michael Schroeder]

It just needs to be removed, the MAP_SHARED was the next argument of the mmap 
call. Force-pushed new version.

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/895#issuecomment-542597491___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Multiple ndb improvements (#895)

[Michael Schroeder]

@mlschroe pushed 1 commit.

b475725264bbe776c9384a22b7c5438dfe8da33f  Refactor mmap/munmap/mremap handling 
in ndb


-- 
You are receiving this because you are subscribed to this thread.
View it on GitHub:
https://github.com/rpm-software-management/rpm/pull/895/files/e716bd973be5ed76c179077f94f629d42711fe9a..b475725264bbe776c9384a22b7c5438dfe8da33f
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Revert "Fully shutdown DBUS on systemd_inhibit cleanup (RhBug:1714657)" (#900)

[Panu Matilainen]

Turns out this isn't a safe thing to do, as an API user could have
their own dbus connections in the same process and shutting those
down is a rather impolite thing to do (and causes crash, burn and
other injuries, eg RhBug:1750575)

This reverts commit d5f201345f6d27b6280750e5c6502f4418614fbc.
You can view, comment on, or merge this pull request online at:

  https://github.com/rpm-software-management/rpm/pull/900

-- Commit Summary --

  * Revert "Fully shutdown DBUS on systemd_inhibit cleanup 
(RhBug:1714657)"

-- File Changes --

M plugins/systemd_inhibit.c (6)

-- Patch Links --

https://github.com/rpm-software-management/rpm/pull/900.patch
https://github.com/rpm-software-management/rpm/pull/900.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/900
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

Re: [Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[Panu Matilainen]

For further background on the topic:

The world is full of key-value databases, but there are *very* few databases 
that support the kind of multiprocess concurrency that rpm requires, and even 
fewer that are license compatible. There's LMDB, but existing versions have 
severe limitations to key size (far below common PATH_MAX) which makes it 
unsuitable for rpm production use. Their upstream promised to up the limit over 
three years ago but it still hasn't happened and we ran out of time five years 
ago. There's something known as WiredTiger which is supposed to be the 
successor to BDB and according to spec sheet is technically suitable, but it's 
hardly time-proven, nobody has ever heard of it and its license is incompatible 
(GPLv3). From there on it gets even more exotic and incompatible.

And then there's sqlite, which obviously is not a key-value store but can serve 
the needs of rpm just fine. It's been there all these years, it's ubiquitous 
and has just the kind of track record we're looking for, but it's been 
overlooked because, well, we already had an sqlite backend and ripped it off, 
so there must be something wrong with it, right? There are couple of important 
factors here:
- The old sqlite backend in the 4.4.x days emulated bdb in a very literal way, 
using multiple separate whole databases for the index tables and all, so it 
couldn't do what it does best *at all*.
- The old sqlite backend was also limited concurrency-wise, because back then 
sqlite didn't have WAL support yet. These days, the amount of concurrency 
sqlite supports is *just* enough for rpm's needs.
- Third, the old sqlite backend did chroot in/out from the database code. That 
is what commit 0508e9a6e3311ed00c34887fd7715d3b469cdca6 is all about - to 
eliminate the need for that. In reality, BDB needs that just as much, it 
currently only works through the smallest of slightly lucky margins, and to do 
transactions on BDB level would require the same thing (which is something we 
could do now too, but that's another topic)

So in the end, going (back) to sqlite was not much of choice, it simply was the 
last database standing: no arbitrary key limitations, multiprocess concurrency, 
proven track record and stable file format, good performance, sane API etc. It 
also has one *huge* bonus over the key-value alternatives: you can actually 
inspect the database contents interactively while sitting in your lazy chair. 
So it's actually rather pleasant to work with. (however there's also the danger 
that it's *too* nice to work with and people will start poking directly into 
the sql instead of going through librpm)

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899#issuecomment-542578228___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint

[Rpm-maint] [rpm-software-management/rpm] Add an sqlite based rpmdb backend (experimental) (#899)

[Panu Matilainen]

All normal functionality is expected to work. Automatic generation
of missing index tables is missing, but that's not relevant at this time.
Going forward, we'll also want some sort of compatibility tracking
for the sql schema.

The database scheme basically just mirrors what BDB does, using strings
for strings and blobs for everything else due to the way integers are
handled in the sqlite C API, for now at least.

Performance is similar or better with BDB in the current unsafe CDB
model, but sqlite uses proper database transactions so this is expected
to be an order of magnitude more robust.

Many things are stupid and/or kind of backwards here due to the internal
API, which I've avoided changing in order to keep it backportable for the
time being. https://github.com/rpm-software-management/rpm/pull/836 is
needed but otherwise this should drop quite trivially into 4.14.x too.
However as we're planning for a longer term future here, it would be dumb
to limit ourselves by what's possible with an internal BDB-oriented API,
so I've fairly major changes planned in that direction.

Note that this changes the default backend for the sake of testing
it in the CI, that will need to be removed before merging, the time to
talk about default changes is (much) later.
You can view, comment on, or merge this pull request online at:

  https://github.com/rpm-software-management/rpm/pull/899

-- Commit Summary --

  * Add an sqlite based rpmdb backend (experimental)

-- File Changes --

M configure.ac (23)
M lib/Makefile.am (6)
M lib/backend/dbi.c (14)
M lib/backend/dbi.h (5)
A lib/backend/sqlite.c (605)
M macros.in (5)

-- Patch Links --

https://github.com/rpm-software-management/rpm/pull/899.patch
https://github.com/rpm-software-management/rpm/pull/899.diff

-- 
You are receiving this because you are subscribed to this thread.
Reply to this email directly or view it on GitHub:
https://github.com/rpm-software-management/rpm/pull/899
___
Rpm-maint mailing list
Rpm-maint@lists.rpm.org
http://lists.rpm.org/mailman/listinfo/rpm-maint