Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

* Howard Chu:

> 5) BerkeleyDB lives under a licensing cloud. Since 2008 Oracle
> lawyers were contacting commercial OpenLDAP users and demanding
> license fees from them, even though BerkeleyDB is expressly licensed
> for free use in open source software (such as OpenLDAP).

At least the new license is explicitly compatible with the GPL,
version 3.  It's easy to read the old Sleeypcat license in such a way
that is incompatible because it imposes an additional restriction: It
offers nothing comparable to the GPL's development/cloud computing
exception.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Jan Zelený]

On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote:
> FWIW I found the new database backend (not mentioned anywhere in the
> original submission).  It is here:
> 
>   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
> 
> Since this change has (rashly) been approved by FESCO, I guess we're
> going to be copying this code into some libguestfs tools, and we'll
> have to keep up with changes to RPM.  It's a drag compared to using a
> real key-value store.

May I ask why? Accessing rpm database in a different way than through librpm 
API is highly discouraged even now.

Thanks
Jan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Mon, Feb 01, 2016 at 09:44:29AM +0100, Jan Zelený wrote:
> On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote:
> > FWIW I found the new database backend (not mentioned anywhere in the
> > original submission).  It is here:
> > 
> >   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
> > 
> > Since this change has (rashly) been approved by FESCO, I guess we're
> > going to be copying this code into some libguestfs tools, and we'll
> > have to keep up with changes to RPM.  It's a drag compared to using a
> > real key-value store.
> 
> May I ask why? Accessing rpm database in a different way than through librpm 
> API is highly discouraged even now.

It's discussed in this thread already.  Please see:

https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/QZG6BVLOOYYGLFJQK5RQ5LAPJIL2KMYF/
https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/NYEVENDEUVPRMWD7PJM2IJSR54H25SKI/

Even if the RPM database is only accessed via librpm, it's still
important that the most central database present on every Fedora
system is reliable, well-tested and flexible.  Sqlite is a highly
regarded piece of software, which runs on billions of Android phones.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

On 02/01/2016 11:43 AM, Richard W.M. Jones wrote:
> On Mon, Feb 01, 2016 at 11:39:33AM +0100, Florian Weimer wrote:
>> On 02/01/2016 11:26 AM, Richard W.M. Jones wrote:
>>> On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote:
 On 02/01/2016 10:59 AM, Richard W.M. Jones wrote:

> Even if the RPM database is only accessed via librpm, it's still
> important that the most central database present on every Fedora
> system is reliable, well-tested and flexible.  Sqlite is a highly
> regarded piece of software, which runs on billions of Android phones.

 That's a strange argument because SQLite is optimized for the mobile use
 case, where durability is not a concern:

   
>>>
>>> I don't see that this posting backs up what you are saying.  Of course
>>> databases don't perform the impossible - you have to wait for the
>>> commit to succeed before the data is persisted, anything else requires
>>> backup power supplies and redundant disks.  But sqlite is much more
>>> likely to be durable than some hand-written brand new code.
>>
>> Please read the thread.  SQLite does not persist the unlink operation on
>> the journal, and an existing valid journal file triggers transaction
>> rollback.
> 
> Can you point to the messages?  I only see two messages in that
> thread, and neither uses the word "unlink".

Sorry, the rest of the thread is here:

  

I expect SQLITE_EXTRA_DURABLE=1 will be the default, considering that
XFS and Ceph both need it to get durability.

In the unlikely scenario that upstream does not change their opinion,
we'll should still enable it by default, at least for synchronous=full
mode.  Curiously, the documentation of pragma SYNCHRONOUS is completely
correct—it only talks about preserving database integrity, not about
preserving the last committed transaction.

> It's good that sqlite is having discussions about durability however.
> One thing we can be sure is that only a minute fraction of that
> developer effort will be spent examining problems in the new RPM
> database code.

On the other hand, the RPM locking requirements are somewhat peculiar,
and it's not clear whether SQLite is a good match.

Florian
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

On 02/01/2016 10:59 AM, Richard W.M. Jones wrote:

> Even if the RPM database is only accessed via librpm, it's still
> important that the most central database present on every Fedora
> system is reliable, well-tested and flexible.  Sqlite is a highly
> regarded piece of software, which runs on billions of Android phones.

That's a strange argument because SQLite is optimized for the mobile use
case, where durability is not a concern:

  

On the other hand, RPM is not crash-safe at all at present, so this
SQLite bug would not prevent its use in RPM.

Florian
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

On 02/01/2016 11:26 AM, Richard W.M. Jones wrote:
> On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote:
>> On 02/01/2016 10:59 AM, Richard W.M. Jones wrote:
>>
>>> Even if the RPM database is only accessed via librpm, it's still
>>> important that the most central database present on every Fedora
>>> system is reliable, well-tested and flexible.  Sqlite is a highly
>>> regarded piece of software, which runs on billions of Android phones.
>>
>> That's a strange argument because SQLite is optimized for the mobile use
>> case, where durability is not a concern:
>>
>>   
> 
> I don't see that this posting backs up what you are saying.  Of course
> databases don't perform the impossible - you have to wait for the
> commit to succeed before the data is persisted, anything else requires
> backup power supplies and redundant disks.  But sqlite is much more
> likely to be durable than some hand-written brand new code.

Please read the thread.  SQLite does not persist the unlink operation on
the journal, and an existing valid journal file triggers transaction
rollback.

Florian
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Mon, Feb 01, 2016 at 11:39:33AM +0100, Florian Weimer wrote:
> On 02/01/2016 11:26 AM, Richard W.M. Jones wrote:
> > On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote:
> >> On 02/01/2016 10:59 AM, Richard W.M. Jones wrote:
> >>
> >>> Even if the RPM database is only accessed via librpm, it's still
> >>> important that the most central database present on every Fedora
> >>> system is reliable, well-tested and flexible.  Sqlite is a highly
> >>> regarded piece of software, which runs on billions of Android phones.
> >>
> >> That's a strange argument because SQLite is optimized for the mobile use
> >> case, where durability is not a concern:
> >>
> >>   
> > 
> > I don't see that this posting backs up what you are saying.  Of course
> > databases don't perform the impossible - you have to wait for the
> > commit to succeed before the data is persisted, anything else requires
> > backup power supplies and redundant disks.  But sqlite is much more
> > likely to be durable than some hand-written brand new code.
> 
> Please read the thread.  SQLite does not persist the unlink operation on
> the journal, and an existing valid journal file triggers transaction
> rollback.

Can you point to the messages?  I only see two messages in that
thread, and neither uses the word "unlink".

It's good that sqlite is having discussions about durability however.
One thing we can be sure is that only a minute fraction of that
developer effort will be spent examining problems in the new RPM
database code.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote:
> On 02/01/2016 10:59 AM, Richard W.M. Jones wrote:
> 
> > Even if the RPM database is only accessed via librpm, it's still
> > important that the most central database present on every Fedora
> > system is reliable, well-tested and flexible.  Sqlite is a highly
> > regarded piece of software, which runs on billions of Android phones.
> 
> That's a strange argument because SQLite is optimized for the mobile use
> case, where durability is not a concern:
> 
>   

I don't see that this posting backs up what you are saying.  Of course
databases don't perform the impossible - you have to wait for the
commit to succeed before the data is persisted, anything else requires
backup power supplies and redundant disks.  But sqlite is much more
likely to be durable than some hand-written brand new code.

The new RPM database code - for example - does not fsync the directory
after updating it, meaning that database files can just disappear in
the event of a power failure.  This is a classic bug which I'm sure
will be fixed, but since I found that problem in 5 minutes looking at
the code, it's very likely that many other problems exist.

> On the other hand, RPM is not crash-safe at all at present, so this
> SQLite bug would not prevent its use in RPM.

Just because one part of RPM isn't crash-safe doesn't mean no part of
it should be.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Mon, Feb 01, 2016 at 10:26:34AM +, Richard W.M. Jones wrote:
> On Mon, Feb 01, 2016 at 11:13:08AM +0100, Florian Weimer wrote:
> > On 02/01/2016 10:59 AM, Richard W.M. Jones wrote:
> > 
> > > Even if the RPM database is only accessed via librpm, it's still
> > > important that the most central database present on every Fedora
> > > system is reliable, well-tested and flexible.  Sqlite is a highly
> > > regarded piece of software, which runs on billions of Android phones.
> > 
> > That's a strange argument because SQLite is optimized for the mobile use
> > case, where durability is not a concern:
> > 
> >   

BTW, in case people really believe that sqlite is "optimized for the
mobile use case" (which it is not), this is link explains the
properties:

https://www.sqlite.org/transactional.html

And this explains the use cases (which do included embedded devices,
but much else besides):

https://www.sqlite.org/whentouse.html

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

On 01/31/2016 04:17 PM, Jonathan Underwood wrote:
> On 29 Jan 2016 10:03 pm, "Richard W.M. Jones"  wrote:
>>
>>
>> FWIW I found the new database backend (not mentioned anywhere in the
>> original submission).  It is here:
>>
>>   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
>>
>> Since this change has (rashly) been approved by FESCO, I guess we're
>> going to be copying this code into some libguestfs tools, and we'll
>> have to keep up with changes to RPM.  It's a drag compared to using a
>> real key-value store.
>>
>>
> 
> I wonder if the MDB (or LMDB) database from the openldap team was evaluated
> as a possible BDB replacement?

LMDB seems a poor match because it requires the administrator to set a
maximum database size, its key length is limited compared to Berkeley DB
(which supports multi-megabyte keys).  This means integrating it takes
more than just rewriting the API calls.

Florian
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Jan Zelený]

On 1. 2. 2016 at 12:14:18, Richard W.M. Jones wrote:
> On Mon, Feb 01, 2016 at 12:51:47PM +0100, Jan Zelený wrote:
> > On 1. 2. 2016 at 09:59:23, Richard W.M. Jones wrote:
> > > On Mon, Feb 01, 2016 at 09:44:29AM +0100, Jan Zelený wrote:
> > > > On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote:
> > > > > FWIW I found the new database backend (not mentioned anywhere in the
> > > > > 
> > > > > original submission).  It is here:
> > > > >   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
> > > > > 
> > > > > Since this change has (rashly) been approved by FESCO, I guess we're
> > > > > going to be copying this code into some libguestfs tools, and we'll
> > > > > have to keep up with changes to RPM.  It's a drag compared to using
> > > > > a
> > > > > real key-value store.
> > > > 
> > > > May I ask why? Accessing rpm database in a different way than through
> > > > librpm API is highly discouraged even now.
> > > 
> > > It's discussed in this thread already.  Please see:
> > > 
> > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.
> > > org/ message/QZG6BVLOOYYGLFJQK5RQ5LAPJIL2KMYF/
> > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.
> > > org
> > > /message/NYEVENDEUVPRMWD7PJM2IJSR54H25SKI/
> > > 
> > > Even if the RPM database is only accessed via librpm, it's still
> > > important that the most central database present on every Fedora
> > > system is reliable, well-tested and flexible.  Sqlite is a highly
> > > regarded piece of software, which runs on billions of Android phones.
> > 
> > Fair enough but that doesn't answer my question. What is the reason for
> > copying rpm code in libguestfs instead of using librpm?
> 
> I don't have a clear answer on that yet, because I'm still waiting to
> find out if the database format will change incompatibly in future.
> 
> If it changes incompatibly in future, and if librpm won't maintain
> backwards compatibility with old databases, then we'll need to take
> snapshots of the librpm code to ensure we can read old RPM databases
> out of guests.
> 
> Notes:
> 
> - I'm not talking about the BDB format - I understand the licensing
>   issues which require a clean break.
> 
> - I couldn't find a way in the the current librpm code to open the RPM
>   database from a different location - obviously required because we
>   need to open the database using our own trusted code, *not* the
>   guest's untrusted code as was proposed elsewhere in this thread.
>   However that's not a reason not to use librpm, since we could add
>   APIs to do that.

Thanks for the input Rich. We will be discussing this topic at our team 
meeting next Monday, I will make sure your feedback is taken into account and 
acted upon.

Jan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Petr Spacek]

On 1.2.2016 12:05, Florian Weimer wrote:
> On 01/31/2016 04:17 PM, Jonathan Underwood wrote:
>> On 29 Jan 2016 10:03 pm, "Richard W.M. Jones"  wrote:
>>>
>>>
>>> FWIW I found the new database backend (not mentioned anywhere in the
>>> original submission).  It is here:
>>>
>>>   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
>>>
>>> Since this change has (rashly) been approved by FESCO, I guess we're
>>> going to be copying this code into some libguestfs tools, and we'll
>>> have to keep up with changes to RPM.  It's a drag compared to using a
>>> real key-value store.
>>>
>>>
>>
>> I wonder if the MDB (or LMDB) database from the openldap team was evaluated
>> as a possible BDB replacement?
> 
> LMDB seems a poor match because it requires the administrator to set a
> maximum database size, its key length is limited compared to Berkeley DB
> (which supports multi-megabyte keys).  This means integrating it takes
> more than just rewriting the API calls.
> 
> Florian

For the wider audience, some more information about LMDB backend for RPM can
be found in:
https://bugzilla.redhat.com/show_bug.cgi?id=1086784

In short:
- Maximum database size is just "sanity limit" set by application. It can be
hunderds of GB if you wish.

- Working around key length really needs some work, yes. On the other hand, it
is clearly possible because SQLite was ported to LMDB as proof-of-concept, and
was actually faster than original implementation. See the README here:
https://github.com/LMDB/sqlightning

-- 
Petr Spacek  @  Red Hat
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

On 02/01/2016 02:55 PM, Petr Spacek wrote:

> For the wider audience, some more information about LMDB backend for RPM can
> be found in:
> https://bugzilla.redhat.com/show_bug.cgi?id=1086784
> 
> In short:
> - Maximum database size is just "sanity limit" set by application. It can be
> hunderds of GB if you wish.

In the past, this wasn't true for 32-bit architectures because the
database file hatd to be mapped.

> - Working around key length really needs some work, yes. On the other hand, it
> is clearly possible because SQLite was ported to LMDB as proof-of-concept, and
> was actually faster than original implementation. See the README here:
> https://github.com/LMDB/sqlightning

First filed issue:

“indices get corrupted if keys are too big”



Florian
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Chris Adams]

Once upon a time, Neal Gompa  said:
> My understanding of the problem is that it's less about the "doubts
> about its future" and more about the fact Oracle inexplicably changed
> the license with BDB 6.x to AGPLv3. Berkley DB 5.3 is old, and no one
> has forked it and maintained it.

To me, it seems like it would be less work to maintain an already
implemented database (BDB) than to start from scratch.  I understand
that nobody is happy with Oracle and their license change, but that
doesn't affect the version already in use in many places.  If even a
couple of the current BDB users (say, OpenLDAP and RPM) had stepped up,
it probably wouldn't be that much work, but instead, everybody seems
intent on re-inventing their own wheels (and spending more effort in the
process).

-- 
Chris Adams 
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Howard Chu]

Chris Adams  cmadams.net> writes:

> 
> Once upon a time, Neal Gompa  gmail.com> said:
> > My understanding of the problem is that it's less about the "doubts
> > about its future" and more about the fact Oracle inexplicably changed
> > the license with BDB 6.x to AGPLv3. Berkley DB 5.3 is old, and no one
> > has forked it and maintained it.
> 
> To me, it seems like it would be less work to maintain an already
> implemented database (BDB) than to start from scratch.  I understand
> that nobody is happy with Oracle and their license change, but that
> doesn't affect the version already in use in many places.  If even a
> couple of the current BDB users (say, OpenLDAP and RPM) had stepped up,
> it probably wouldn't be that much work, but instead, everybody seems
> intent on re-inventing their own wheels (and spending more effort in the
> process).
> 
We "reinvented the wheel" with LMDB for multiple reasons, both technical and
political.

1) BerkeleyDB is a large code base. 1.7MLOCs
https://www.openhub.net/p/berkeleydb . LMDB is only 7KLOCs
2) BerkeleyDB includes a large number of features that are irrelevant to
OpenLDAP (hash DBs, replication, multiple other access methods). LMDB is
strictly a B+tree.
3) BerkeleyDB is slow, and requires extensive, complex tuning to wring any
glimmer of performance out of it. LMDB requires no tuning and is orders of
magnitude faster.
4) BerkeleyDB is fragile, and even a slight misconfiguration can render its
transaction logs useless for crash recovery. LMDB is crash-proof by design,
and that design has held up in the real world.

5) BerkeleyDB lives under a licensing cloud. Since 2008 Oracle lawyers were
contacting commercial OpenLDAP users and demanding license fees from them,
even though BerkeleyDB is expressly licensed for free use in open source
software (such as OpenLDAP). Even if we forked BerkeleyDB and maintained it
ourselves, Oracle's lawyers would continue to illegitimately harass OpenLDAP
users.

LMDB has been in use since 2011. At this point it has saved tremendous
administration and maintenance cost, both for us (Symas and OpenLDAP
Project) and for all the other projects that have adopted it.
-- 
  -- Howard Chu
  CTO, Symas Corp.   http://www.symas.com
  Director, Highland Sun http://highlandsun.com/hyc/
  Chief Architect, OpenLDAP  http://www.openldap.org/project/
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Jan Zelený]

On 1. 2. 2016 at 09:59:23, Richard W.M. Jones wrote:
> On Mon, Feb 01, 2016 at 09:44:29AM +0100, Jan Zelený wrote:
> > On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote:
> > > FWIW I found the new database backend (not mentioned anywhere in the
> > > 
> > > original submission).  It is here:
> > >   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
> > > 
> > > Since this change has (rashly) been approved by FESCO, I guess we're
> > > going to be copying this code into some libguestfs tools, and we'll
> > > have to keep up with changes to RPM.  It's a drag compared to using a
> > > real key-value store.
> > 
> > May I ask why? Accessing rpm database in a different way than through
> > librpm API is highly discouraged even now.
> 
> It's discussed in this thread already.  Please see:
> 
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/
> message/QZG6BVLOOYYGLFJQK5RQ5LAPJIL2KMYF/
> https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> /message/NYEVENDEUVPRMWD7PJM2IJSR54H25SKI/
> 
> Even if the RPM database is only accessed via librpm, it's still
> important that the most central database present on every Fedora
> system is reliable, well-tested and flexible.  Sqlite is a highly
> regarded piece of software, which runs on billions of Android phones.

Fair enough but that doesn't answer my question. What is the reason for 
copying rpm code in libguestfs instead of using librpm?

Thanks
Jan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Mon, Feb 01, 2016 at 12:51:47PM +0100, Jan Zelený wrote:
> On 1. 2. 2016 at 09:59:23, Richard W.M. Jones wrote:
> > On Mon, Feb 01, 2016 at 09:44:29AM +0100, Jan Zelený wrote:
> > > On 29. 1. 2016 at 22:03:00, Richard W.M. Jones wrote:
> > > > FWIW I found the new database backend (not mentioned anywhere in the
> > > > 
> > > > original submission).  It is here:
> > > >   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
> > > > 
> > > > Since this change has (rashly) been approved by FESCO, I guess we're
> > > > going to be copying this code into some libguestfs tools, and we'll
> > > > have to keep up with changes to RPM.  It's a drag compared to using a
> > > > real key-value store.
> > > 
> > > May I ask why? Accessing rpm database in a different way than through
> > > librpm API is highly discouraged even now.
> > 
> > It's discussed in this thread already.  Please see:
> > 
> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/
> > message/QZG6BVLOOYYGLFJQK5RQ5LAPJIL2KMYF/
> > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
> > /message/NYEVENDEUVPRMWD7PJM2IJSR54H25SKI/
> > 
> > Even if the RPM database is only accessed via librpm, it's still
> > important that the most central database present on every Fedora
> > system is reliable, well-tested and flexible.  Sqlite is a highly
> > regarded piece of software, which runs on billions of Android phones.
> 
> Fair enough but that doesn't answer my question. What is the reason for 
> copying rpm code in libguestfs instead of using librpm?

I don't have a clear answer on that yet, because I'm still waiting to
find out if the database format will change incompatibly in future.

If it changes incompatibly in future, and if librpm won't maintain
backwards compatibility with old databases, then we'll need to take
snapshots of the librpm code to ensure we can read old RPM databases
out of guests.

Notes:

- I'm not talking about the BDB format - I understand the licensing
  issues which require a clean break.

- I couldn't find a way in the the current librpm code to open the RPM
  database from a different location - obviously required because we
  need to open the database using our own trusted code, *not* the
  guest's untrusted code as was proposed elsewhere in this thread.
  However that's not a reason not to use librpm, since we could add
  APIs to do that.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Chris Adams]

Once upon a time, Jonathan Underwood  said:
> I wonder if the MDB (or LMDB) database from the openldap team was evaluated
> as a possible BDB replacement?

There's also TDB from Samba.

I certainly don't know RPM's requirements, but it seems shortsighted
from a long-term maintenance point of view to implement a custom
database, both from a format and a code point of view.

The prime reason to change away from BDB is "doubts about its future and
level of maintenance" - how is reinventing a wheel any easier than
keeping an existing wheel working?

-- 
Chris Adams 
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Neal Gompa]

On Sun, Jan 31, 2016 at 8:36 PM, Chris Adams  wrote:
> Once upon a time, Jonathan Underwood  said:
>> I wonder if the MDB (or LMDB) database from the openldap team was evaluated
>> as a possible BDB replacement?
>
> There's also TDB from Samba.
>
> I certainly don't know RPM's requirements, but it seems shortsighted
> from a long-term maintenance point of view to implement a custom
> database, both from a format and a code point of view.
>
> The prime reason to change away from BDB is "doubts about its future and
> level of maintenance" - how is reinventing a wheel any easier than
> keeping an existing wheel working?
>

My understanding of the problem is that it's less about the "doubts
about its future" and more about the fact Oracle inexplicably changed
the license with BDB 6.x to AGPLv3. Berkley DB 5.3 is old, and no one
has forked it and maintained it. That said, my guess is that they
viewed the change as an opportunity to deal with some long-standing
issues with the current RPM Database format and system, as they've
detailed on the rpm.org site[0]. I'm not sure to what extent these
problems are true now, but that may have been one of the goals of the
new database format.

[0]: http://rpm.org/wiki/Problems/RpmDB

-- 
真実はいつも一つ！/ Always, there's only one truth!
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Jonathan Underwood]

On 29 Jan 2016 10:03 pm, "Richard W.M. Jones"  wrote:
>
>
> FWIW I found the new database backend (not mentioned anywhere in the
> original submission).  It is here:
>
>   http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb
>
> Since this change has (rashly) been approved by FESCO, I guess we're
> going to be copying this code into some libguestfs tools, and we'll
> have to keep up with changes to RPM.  It's a drag compared to using a
> real key-value store.
>
>

I wonder if the MDB (or LMDB) database from the openldap team was evaluated
as a possible BDB replacement?
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Fri, Jan 29, 2016 at 05:18:03PM +0100, Florian Festi wrote:
> On 01/29/2016 05:02 PM, Richard W.M. Jones wrote:
> > I'm still unclear why you don't just use sqlite 3.
> 
> Among other things it is about 2 times lower than the current BDB
> implementation.

There's some question [see adjacent message in this thread] about
whether that was measuring sqlite 2 or 3.

Will a custom database implementation be any faster?  And will it
solve all the other problems, such as reliability, flexibility,
backwards compatibility, that sqlite already solves?  Sqlite gets this
stuff right.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/14/2016 01:08 PM, Richard W.M. Jones wrote:
> Yup - I'm curious about this as well.  Using sqlite [for example]
> would solve the libguestfs issues I outlined in this thread, as well
> as using a format which is robust and proven rather than some
> half-baked homebrew thing.

Well, we actually deleted a sqlite backend a few years ago. But even
with a more relational DB layout performance is about 2 times worse than
the BDB.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Weimer]

On 01/29/2016 04:35 PM, Florian Festi wrote:
> On 01/14/2016 01:08 PM, Richard W.M. Jones wrote:
>> Yup - I'm curious about this as well.  Using sqlite [for example]
>> would solve the libguestfs issues I outlined in this thread, as well
>> as using a format which is robust and proven rather than some
>> half-baked homebrew thing.
> 
> Well, we actually deleted a sqlite backend a few years ago. But even
> with a more relational DB layout performance is about 2 times worse than
> the BDB.

Wasn't it still using SQLite 2?

Florian
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/13/2016 04:04 PM, Daniel P. Berrange wrote:
> On Wed, Jan 13, 2016 at 01:30:59PM +, Richard Hughes wrote:
>> On 13 January 2016 at 13:13, Reindl Harald  wrote:
>>> so there is no justification to declare one need to install from scratch
>>> just because rpm which works for many years fine changes it's storage format
>>
>> I don't think anyone said there was a need to reinstall from scratch.
> 
> Well the feature writeup is rather fuzzy on this.  It says that in Fedora
> 24 rpm will be able to read both old and new format, but it also says
> that future RPM versions will drop support for the old format.

Yes, this is exactly what I meant to say.

> So unless
> there is a mandatory data format conversion during some Fedora upgrade,
> then at some point RPM will cease to be able to read existing installs
> with the old format which could imply a need to reinstall.

Yes, dropping the support for the old format will be a separate System
wide change for some future Fedora release. Looking at all those
questions about managing containers and VMs we might take a longer time
than our BDB folks may like.

> IMHO the feature proposal text needs to be more explicit about the upgrade
> path and exactly when any data conversion will take place, to avoid leaving
> existing installs with old format stuck with Fedora 25 rpm (or later) drops
> BDB support entirely.

Well, the update path that will always will be available is doing a
rpmdb --exportdb with the old rpm version and an rpmdb --importdb with
the new version. The format of the rpm header will stay compatible so
getting them out and then back into the DB solves the problem.

Florian


-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/16/2016 02:33 AM, Kevin Kofler wrote:
> Daniel P. Berrange wrote:
>> Well the feature writeup is rather fuzzy on this.  It says that in Fedora
>> 24 rpm will be able to read both old and new format, but it also says
>> that future RPM versions will drop support for the old format. So unless
>> there is a mandatory data format conversion during some Fedora upgrade,
>> then at some point RPM will cease to be able to read existing installs
>> with the old format which could imply a need to reinstall.
> 
> Will rpm --rebuilddb convert the database?

Yes.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Fri, Jan 29, 2016 at 04:53:08PM +0100, Florian Festi wrote:
> On 01/13/2016 03:07 PM, Richard W.M. Jones wrote:
> > 
> > Say, for example, that Fedora 24 moves to the new format.  Will Fedora 34
> > be able to read Fedora 24 RPM databases?
> 
> Fedora 25 will. Fedora 34 will probably not. If it turns out that
> reading 10 releases old database is really, really important. We will
> either keep the bdb support or offer some tools to basically do a rpmdb
> --exportdb which can be used to create a modern rpm database. Who is
> maintaining the BDB needed for this in 2021 is not a question I am going
> to answer, though.

I don't think I was clear.  I understand that for licensing reasons
BDB is being dropped.  My question relates entirely to this new
format, and is nothing to do with BDB.

Will the new format be forwards and backwards compatible?

I'm still unclear why you don't just use sqlite 3.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
Fedora Windows cross-compiler. Compile Windows programs, test, and
build Windows installers. Over 100 libraries supported.
http://fedoraproject.org/wiki/MinGW
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/29/2016 05:02 PM, Richard W.M. Jones wrote:
> On Fri, Jan 29, 2016 at 04:53:08PM +0100, Florian Festi wrote:
>> On 01/13/2016 03:07 PM, Richard W.M. Jones wrote:
>>>
>>> Say, for example, that Fedora 24 moves to the new format.  Will Fedora 34
>>> be able to read Fedora 24 RPM databases?
>>
>> Fedora 25 will. Fedora 34 will probably not. If it turns out that
>> reading 10 releases old database is really, really important. We will
>> either keep the bdb support or offer some tools to basically do a rpmdb
>> --exportdb which can be used to create a modern rpm database. Who is
>> maintaining the BDB needed for this in 2021 is not a question I am going
>> to answer, though.
> 
> I don't think I was clear.  I understand that for licensing reasons
> BDB is being dropped.  My question relates entirely to this new
> format, and is nothing to do with BDB.
> 
> Will the new format be forwards and backwards compatible?

I really hope we do not have to change the format again in the
foreseeable future. At least I can imagine other things to do with my
life than changing the RPM DB format over and over.

> I'm still unclear why you don't just use sqlite 3.

Among other things it is about 2 times lower than the current BDB
implementation.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/29/2016 05:15 PM, Dennis Gilmore wrote:
> On Monday, January 11, 2016 03:46:27 PM Jan Kurik wrote:
>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
>>
>> Change owner(s):
>> * Florian Festi < ffesti AT redhat DOT com >
>>
>>
>> Change format of the RPM Database from Berkeley DB to RPM's own format.
>>
>> == Detailed Description ==
>> The current implementation of the RPM Database is based on Berkeley
>> DB. There are doubts about the its future and level of maintenance. In
>> addition rpm's use of the database has multiple issues on its own. As
>> a result RPM upstream is working to replace the database format with a
>> new implementation.
> How will this work in situations like the builders. Rpm on f23 makes the f24 
> chroots. I am guessing that will be mostly ok, the db will just have the old 
> format. But what about when we move the builders to f24 and use f24 and rpm 
> with the new db format to make rhel 5, 6, 7 and Fedora, 22 and 23 buildroots. 
> Is there the potential that things will break horribly?

Yes, there is. So this needs to be managed properly. By setting the
format of those databases or by converting them to the old format. Even
if the builder were not to be able to create the old format you could
still use rpmdb --export from outside the buildroot to get the headers
out of the DB and then use rpmdb --importdb from within to rebuild the
old format.

In general the database format is not as important as one might think as
the header format is what carries the actual data.

Florian


-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/13/2016 03:07 PM, Richard W.M. Jones wrote:
> 
> Say, for example, that Fedora 24 moves to the new format.  Will Fedora 34
> be able to read Fedora 24 RPM databases?

Fedora 25 will. Fedora 34 will probably not. If it turns out that
reading 10 releases old database is really, really important. We will
either keep the bdb support or offer some tools to basically do a rpmdb
--exportdb which can be used to create a modern rpm database. Who is
maintaining the BDB needed for this in 2021 is not a question I am going
to answer, though.

> How about Fedora 24 being
> able to read Fedora 34 RPM databases?

I hope we don't have to change the format more often than once a decade.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Dennis Gilmore]

On Monday, January 11, 2016 03:46:27 PM Jan Kurik wrote:
> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
> 
> Change owner(s):
> * Florian Festi < ffesti AT redhat DOT com >
> 
> 
> Change format of the RPM Database from Berkeley DB to RPM's own format.
> 
> == Detailed Description ==
> The current implementation of the RPM Database is based on Berkeley
> DB. There are doubts about the its future and level of maintenance. In
> addition rpm's use of the database has multiple issues on its own. As
> a result RPM upstream is working to replace the database format with a
> new implementation.
How will this work in situations like the builders. Rpm on f23 makes the f24 
chroots. I am guessing that will be mostly ok, the db will just have the old 
format. But what about when we move the builders to f24 and use f24 and rpm 
with the new db format to make rhel 5, 6, 7 and Fedora, 22 and 23 buildroots. 
Is there the potential that things will break horribly?

Dennis

signature.asc
Description: This is a digitally signed message part.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Wed, Jan 13, 2016 at 02:07:57PM +, Richard W.M. Jones wrote:
> Another thing to think about is endianness and word size, since with
> BDB we can examine, say, an i686 guest from an ppc64 host (and people
> even do this).

Also what about this part of my question?

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-p2v converts physical machines to virtual machines.  Boot with a
live CD or over the network (PXE) and turn machines into KVM guests.
http://libguestfs.org/virt-v2v
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

FWIW I found the new database backend (not mentioned anywhere in the
original submission).  It is here:

  http://rpm.org/gitweb?p=rpm.git;a=tree;f=lib/backend/ndb

Since this change has (rashly) been approved by FESCO, I guess we're
going to be copying this code into some libguestfs tools, and we'll
have to keep up with changes to RPM.  It's a drag compared to using a
real key-value store.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Przemek Klosowski]

On 01/29/2016 11:18 AM, Florian Festi wrote:

On 01/29/2016 05:02 PM, Richard W.M. Jones wrote:

I'm still unclear why you don't just use sqlite 3.

Among other things it is about 2 times lower than the current BDB
implementation.
As a third party in all this, I'd rather talent be applied to speeding 
up sqlite3 than to writing a new database engine. It would be a win-win!
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Jeffrey Ollie]

On Fri, Jan 15, 2016 at 2:24 AM, Tomáš Smetana  wrote:

>
> I tend to use systemd-nspawn containers for building rpms. So for example,
> I
> have a Fedora 24 system and use its dnf to create e.g. Centos 7 container
> root and then build Centos rpms from within that container.  If I
> understand
> the change correctly, this is going to break since the Centos 7 rpm-build
> will not be able to read the database created by the Fedora 24 dnf.
>

I personally prefer to build my containers by starting with the raw files
that are used to build the base CentOS and Fedora Docker containers:

https://raw.githubusercontent.com/CentOS/sig-cloud-instance-images/CentOS-7.2.1511/docker/centos-7.2.1511-docker.tar.xz
https://download.fedoraproject.org/pub/fedora/linux/releases/23/Docker/x86_64/Fedora-Docker-Base-23-20151030.x86_64.tar.xz

The CentOS tarfile can be easily untarred into an empty directory.  The
Fedora tarball contains a layer.tar file that contains the raw filesystem.

I like doing it this way because I don't have to depend on the host yum/dnf
to set up the filesystem.  This would work too for non-rpm systems (or for
RHEL) as well but I haven't had the need so I haven't searched for the
appropriate media.

-- 
Jeff Ollie
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Kevin Kofler]

Daniel P. Berrange wrote:
> Well the feature writeup is rather fuzzy on this.  It says that in Fedora
> 24 rpm will be able to read both old and new format, but it also says
> that future RPM versions will drop support for the old format. So unless
> there is a mandatory data format conversion during some Fedora upgrade,
> then at some point RPM will cease to be able to read existing installs
> with the old format which could imply a need to reinstall.

Will rpm --rebuilddb convert the database?

Kevin Kofler
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Dan Horák]

On Fri, 15 Jan 2016 09:24:36 +0100
Tomáš Smetana  wrote:

> On Wed, 13 Jan 2016 14:38:08 +0100
> Florian Festi  wrote:
> 
> > On 01/13/2016 02:36 PM, Reindl Harald wrote:
> > > 
> > > 
> > > Am 13.01.2016 um 14:30 schrieb Richard Hughes:  
> > >> On 13 January 2016 at 13:13, Reindl Harald
> > >>  wrote:  
> > >>> so there is no justification to declare one need to install
> > >>> from scratch just because rpm which works for many years fine
> > >>> changes it's storage format  
> > >>
> > >> I don't think anyone said there was a need to reinstall from
> > >> scratch  
> > > 
> > > so how do you translate "clearly not forward compatible"?  
> > 
> > "forward compatible" means the old version of a program being able
> > to read/process newer version data.
> > 
> > The current rpm versions will not be able to read the new database
> > format.
> 
> I tend to use systemd-nspawn containers for building rpms. So for
> example, I have a Fedora 24 system and use its dnf to create e.g.
> Centos 7 container root and then build Centos rpms from within that
> container.  If I understand the change correctly, this is going to
> break since the Centos 7 rpm-build will not be able to read the
> database created by the Fedora 24 dnf.
> 
> I know more people using dnf/rpm to "manage" the containers and this
> is somewhat a regression for us.  I'm not sure there is a way to
> prevent this breakage... So just FYI. :)

won't regular mock chroot have the same problem?


Dan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Tomáš Smetana]

On Wed, 13 Jan 2016 14:38:08 +0100
Florian Festi  wrote:

> On 01/13/2016 02:36 PM, Reindl Harald wrote:
> > 
> > 
> > Am 13.01.2016 um 14:30 schrieb Richard Hughes:  
> >> On 13 January 2016 at 13:13, Reindl Harald 
> >> wrote:  
> >>> so there is no justification to declare one need to install from scratch
> >>> just because rpm which works for many years fine changes it's storage
> >>> format  
> >>
> >> I don't think anyone said there was a need to reinstall from scratch  
> > 
> > so how do you translate "clearly not forward compatible"?  
> 
> "forward compatible" means the old version of a program being able to
> read/process newer version data.
> 
> The current rpm versions will not be able to read the new database format.

I tend to use systemd-nspawn containers for building rpms. So for example, I
have a Fedora 24 system and use its dnf to create e.g. Centos 7 container
root and then build Centos rpms from within that container.  If I understand
the change correctly, this is going to break since the Centos 7 rpm-build
will not be able to read the database created by the Fedora 24 dnf.

I know more people using dnf/rpm to "manage" the containers and this is
somewhat a regression for us.  I'm not sure there is a way to prevent this
breakage... So just FYI. :)

Thanks and regards,
-- 
Tomáš Smetana
Platform Engineering, Red Hat
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Jan Kratochvil]

On Fri, 15 Jan 2016 09:42:17 +0100, Dan Horák wrote:
> On Fri, 15 Jan 2016 09:24:36 +0100 Tomáš Smetana  wrote:
> > On Wed, 13 Jan 2016 14:38:08 +0100 Florian Festi  wrote:
> > I tend to use systemd-nspawn containers for building rpms. So for
> > example, I have a Fedora 24 system and use its dnf to create e.g.
> > Centos 7 container root and then build Centos rpms from within that
> > container.  If I understand the change correctly, this is going to
> > break since the Centos 7 rpm-build will not be able to read the
> > database created by the Fedora 24 dnf.
> > 
> > I know more people using dnf/rpm to "manage" the containers and this
> > is somewhat a regression for us.  I'm not sure there is a way to
> > prevent this breakage... So just FYI. :)
> 
> won't regular mock chroot have the same problem?

Yes, it has - for CentOS 6 in recent Fedoras.  Using 'rpmbuild --nodeps' for
that.


Jan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Michael Šimáček]

On 2016-01-15 09:42, Dan Horák wrote:

On Fri, 15 Jan 2016 09:24:36 +0100
Tomáš Smetana  wrote:


On Wed, 13 Jan 2016 14:38:08 +0100
Florian Festi  wrote:


On 01/13/2016 02:36 PM, Reindl Harald wrote:



Am 13.01.2016 um 14:30 schrieb Richard Hughes:

On 13 January 2016 at 13:13, Reindl Harald
 wrote:

so there is no justification to declare one need to install
from scratch just because rpm which works for many years fine
changes it's storage format


I don't think anyone said there was a need to reinstall from
scratch


so how do you translate "clearly not forward compatible"?


"forward compatible" means the old version of a program being able
to read/process newer version data.

The current rpm versions will not be able to read the new database
format.


I tend to use systemd-nspawn containers for building rpms. So for
example, I have a Fedora 24 system and use its dnf to create e.g.
Centos 7 container root and then build Centos rpms from within that
container.  If I understand the change correctly, this is going to
break since the Centos 7 rpm-build will not be able to read the
database created by the Fedora 24 dnf.

I know more people using dnf/rpm to "manage" the containers and this
is somewhat a regression for us.  I'm not sure there is a way to
prevent this breakage... So just FYI. :)


won't regular mock chroot have the same problem?



Mock uses --nodeps when running rpmbuild, because such incompatibilities 
already exist - for example when building EL 6 packages on F23 (i.e. RPM 
from EL6 cannot read the rpmdb in buildroot)


Michael
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Thu, Jan 14, 2016 at 07:50:05AM +0100, Petr Spacek wrote:
> On 13.1.2016 13:48, Florian Festi wrote:
> > On 01/11/2016 03:57 PM, Dan Horák wrote:
> >> On Mon, 11 Jan 2016 15:46:27 +0100
> >> Jan Kurik  wrote:
> >>
> >>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> >>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
> >>>
> >>> Change owner(s):
> >>> * Florian Festi < ffesti AT redhat DOT com >
> >>>
> >>>
> >>> Change format of the RPM Database from Berkeley DB to RPM's own
> >>> format.
> >>>
> >>> == Detailed Description ==
> >>> The current implementation of the RPM Database is based on Berkeley
> >>> DB. There are doubts about the its future and level of maintenance. In
> >>> addition rpm's use of the database has multiple issues on its own. As
> >>> a result RPM upstream is working to replace the database format with a
> >>> new implementation.
> >>
> >> does it mean rpm is changing from Berkeley DB library to another
> >> library or to a completely new implementation of a database engine?
> > 
> > We change to our own format. One of the problem is the bit special
> > requirements of rpm where you want to have non privileged readers that
> > must not have any write access - which is required for most databases
> > for locking.
> 
> I'm curious! Would it be possible to elaborate on reasons why no existing DB
> was good enough for RPM?
> 
> https://bugzilla.redhat.com/show_bug.cgi?id=1086784#c8 indicates that this
> locking requirement is not a problem/obstacle for using LMDB.

Yup - I'm curious about this as well.  Using sqlite [for example]
would solve the libguestfs issues I outlined in this thread, as well
as using a format which is robust and proven rather than some
half-baked homebrew thing.

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-builder quickly builds VMs from scratch
http://libguestfs.org/virt-builder.1.html
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Reindl Harald]

Am 13.01.2016 um 14:01 schrieb Florian Festi:

On 01/11/2016 09:06 PM, Richard W.M. Jones wrote:

On Mon, Jan 11, 2016 at 03:46:27PM +0100, Jan Kurik wrote:

= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
https://fedoraproject.org/wiki/Changes/NewRpmDBFormat


Details of the format?

What forward and backward compatibility guarantees are there?


RPM will keep support for BDB for now. But to get rid of the dependency
it will be dropped at some point in the future. So it will stay backward
compatible.

The file format is clearly not forward compatible (although you will be
for now be able to still use the old format)


"clearly not forward compatible"?

there needs to be a migration path to get a existing bdb rpmdb converted 
our you will blow with "will be dropped at some point in the future" a 
ton of users and admins to hell


we run 30 machines originally installed with Fedora 9 and updated until 
now to F23 which surived UsrMove, migration to grub2 including make 
space for grub2 on the bootdrive with parted and the switch to systemd 
without install them from scratch


so there is no justification to declare one need to install from scratch 
just because rpm which works for many years fine changes it's storage 
format




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard Hughes]

On 13 January 2016 at 13:13, Reindl Harald  wrote:
> so there is no justification to declare one need to install from scratch
> just because rpm which works for many years fine changes it's storage format

I don't think anyone said there was a need to reinstall from scratch.

Richard
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Reindl Harald]

Am 13.01.2016 um 14:30 schrieb Richard Hughes:

On 13 January 2016 at 13:13, Reindl Harald  wrote:

so there is no justification to declare one need to install from scratch
just because rpm which works for many years fine changes it's storage format


I don't think anyone said there was a need to reinstall from scratch


so how do you translate "clearly not forward compatible"?



signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Wed, Jan 13, 2016 at 02:01:23PM +0100, Florian Festi wrote:
> On 01/11/2016 09:06 PM, Richard W.M. Jones wrote:
> > On Mon, Jan 11, 2016 at 03:46:27PM +0100, Jan Kurik wrote:
> >> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> >> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
> > 
> > Details of the format?
> > 
> > What forward and backward compatibility guarantees are there?
> 
> RPM will keep support for BDB for now. But to get rid of the dependency
> it will be dropped at some point in the future. So it will stay backward
> compatible.

What I meant here is what about forward and backward compatibility of
the new format.  Obviously I understand why BDB is being dropped and
that the new format cannot possibly be compatible with BDB.  It's
accepted that there will be a break in Fedora 24 for reasons that are
outside our control.

Say, for example, that Fedora 24 moves to the new format.  Will Fedora 34
be able to read Fedora 24 RPM databases?  How about Fedora 24 being
able to read Fedora 34 RPM databases?

> > Currently we use the (BDB-specific obviously) db_dump tool for this
> > purpose.  Other distros such as Debian as much simpler in this regard
> > since they expose the package data as plain text files.
> > 
> > https://github.com/libguestfs/libguestfs/blob/master/src/inspect-apps.c
> 
> I would advice to change these over to using librpm or one of the rpm
> cli tools. If there are any tools missing, please let me know and we
> will try to come up with them.
>
> If possible the rpm installation of the system examined should be used.

This isn't possible - it's insecure to rely on running guest code
(and not even possible in some situations).

Carrying around old versions of librpm on the host isn't ideal either.

> If this cannot be done you might need a new version of rpm on the host
> system.

Are you implying that all Fedora librpm will be backwards compatible?
(not caring about Fedora < 24 obviously)

Another thing to think about is endianness and word size, since with
BDB we can examine, say, an i686 guest from an ppc64 host (and people
even do this).

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
virt-df lists disk usage of guests without needing to install any
software inside the virtual machine.  Supports Linux and Windows.
http://people.redhat.com/~rjones/virt-df/
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/11/2016 05:26 PM, Kalev Lember wrote:
> On 01/11/2016 03:46 PM, Jan Kurik wrote:
>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
>>
>> Change owner(s):
>> * Florian Festi < ffesti AT redhat DOT com >
>>
>>
>> Change format of the RPM Database from Berkeley DB to RPM's own format.
>>
>> == Detailed Description ==
>> The current implementation of the RPM Database is based on Berkeley
>> DB. There are doubts about the its future and level of maintenance. In
>> addition rpm's use of the database has multiple issues on its own. As
>> a result RPM upstream is working to replace the database format with a
>> new implementation.
> 
> Is the new database going to be able to support yumdb use cases as well?
> Might be a good time to get rid of separate rpmdb and yumdb and merge
> them together.

No, this is a pretty straight forward replacement of the backend. While
the whole area has still a lot the be desired this change only replaces
the BDB. No other fancy changes.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/13/2016 02:36 PM, Reindl Harald wrote:
> 
> 
> Am 13.01.2016 um 14:30 schrieb Richard Hughes:
>> On 13 January 2016 at 13:13, Reindl Harald 
>> wrote:
>>> so there is no justification to declare one need to install from scratch
>>> just because rpm which works for many years fine changes it's storage
>>> format
>>
>> I don't think anyone said there was a need to reinstall from scratch
> 
> so how do you translate "clearly not forward compatible"?

"forward compatible" means the old version of a program being able to
read/process newer version data.

The current rpm versions will not be able to read the new database format.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/11/2016 03:57 PM, Dan Horák wrote:
> On Mon, 11 Jan 2016 15:46:27 +0100
> Jan Kurik  wrote:
> 
>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
>>
>> Change owner(s):
>> * Florian Festi < ffesti AT redhat DOT com >
>>
>>
>> Change format of the RPM Database from Berkeley DB to RPM's own
>> format.
>>
>> == Detailed Description ==
>> The current implementation of the RPM Database is based on Berkeley
>> DB. There are doubts about the its future and level of maintenance. In
>> addition rpm's use of the database has multiple issues on its own. As
>> a result RPM upstream is working to replace the database format with a
>> new implementation.
> 
> does it mean rpm is changing from Berkeley DB library to another
> library or to a completely new implementation of a database engine?

We change to our own format. One of the problem is the bit special
requirements of rpm where you want to have non privileged readers that
must not have any write access - which is required for most databases
for locking.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/11/2016 09:06 PM, Richard W.M. Jones wrote:
> On Mon, Jan 11, 2016 at 03:46:27PM +0100, Jan Kurik wrote:
>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
> 
> Details of the format?
> 
> What forward and backward compatibility guarantees are there?

RPM will keep support for BDB for now. But to get rid of the dependency
it will be dropped at some point in the future. So it will stay backward
compatible.

The file format is clearly not forward compatible (although you will be
for now be able to still use the old format).

> libguestfs reads the rpmdb of guests (of any version, new or old) in
> order to get the list of packages installed in a guest.  Remember that
> the host and guest may be very different versions, eg. a RHEL 9 host
> accessing a Fedora 24 guest.  Or vice versa.

> Currently we use the (BDB-specific obviously) db_dump tool for this
> purpose.  Other distros such as Debian as much simpler in this regard
> since they expose the package data as plain text files.
> 
> https://github.com/libguestfs/libguestfs/blob/master/src/inspect-apps.c

I would advice to change these over to using librpm or one of the rpm
cli tools. If there are any tools missing, please let me know and we
will try to come up with them.

If possible the rpm installation of the system examined should be used.
If this cannot be done you might need a new version of rpm on the host
system.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Petr Spacek]

On 13.1.2016 13:48, Florian Festi wrote:
> On 01/11/2016 03:57 PM, Dan Horák wrote:
>> On Mon, 11 Jan 2016 15:46:27 +0100
>> Jan Kurik  wrote:
>>
>>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
>>>
>>> Change owner(s):
>>> * Florian Festi < ffesti AT redhat DOT com >
>>>
>>>
>>> Change format of the RPM Database from Berkeley DB to RPM's own
>>> format.
>>>
>>> == Detailed Description ==
>>> The current implementation of the RPM Database is based on Berkeley
>>> DB. There are doubts about the its future and level of maintenance. In
>>> addition rpm's use of the database has multiple issues on its own. As
>>> a result RPM upstream is working to replace the database format with a
>>> new implementation.
>>
>> does it mean rpm is changing from Berkeley DB library to another
>> library or to a completely new implementation of a database engine?
> 
> We change to our own format. One of the problem is the bit special
> requirements of rpm where you want to have non privileged readers that
> must not have any write access - which is required for most databases
> for locking.

I'm curious! Would it be possible to elaborate on reasons why no existing DB
was good enough for RPM?

https://bugzilla.redhat.com/show_bug.cgi?id=1086784#c8 indicates that this
locking requirement is not a problem/obstacle for using LMDB.

Thank you for your time!

-- 
Petr Spacek  @  Red Hat
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Florian Festi]

On 01/11/2016 05:08 PM, Colin Walters wrote:
> 
> 
> On Mon, Jan 11, 2016, at 09:46 AM, Jan Kurik wrote:
>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
> 
> It'd be interesting to know the technical details, worth reposting once 
> there's
> a design document or prototype PR.

The format is already in the rpm upstream master branch. Give us a bit
on the documentation.

> I added rpm-ostree to the list, and one thing I'd like to note regarding that:
> We always generate a new DB rather than mutate-in-place, because
> rpm-ostree always generates atomic updates.  This means that
> mutation speed is not very relevant to us, but query speed still
> is.

We did some benchmarks. So far performance seems OK or better.

Florian

-- 

Red Hat GmbH, http://www.de.redhat.com/ Registered seat: Grasbrunn,
Commercial register: Amtsgericht Muenchen, HRB 153243,
Managing Directors: Charles Cachera, Michael Cunningham, Michael
O'Neill, Charles Peters
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Daniel P. Berrange]

On Wed, Jan 13, 2016 at 01:30:59PM +, Richard Hughes wrote:
> On 13 January 2016 at 13:13, Reindl Harald  wrote:
> > so there is no justification to declare one need to install from scratch
> > just because rpm which works for many years fine changes it's storage format
> 
> I don't think anyone said there was a need to reinstall from scratch.

Well the feature writeup is rather fuzzy on this.  It says that in Fedora
24 rpm will be able to read both old and new format, but it also says
that future RPM versions will drop support for the old format. So unless
there is a mandatory data format conversion during some Fedora upgrade,
then at some point RPM will cease to be able to read existing installs
with the old format which could imply a need to reinstall.

IMHO the feature proposal text needs to be more explicit about the upgrade
path and exactly when any data conversion will take place, to avoid leaving
existing installs with old format stuck with Fedora 25 rpm (or later) drops
BDB support entirely.

Regards,
Daniel
-- 
|: http://berrange.com  -o-http://www.flickr.com/photos/dberrange/ :|
|: http://libvirt.org  -o- http://virt-manager.org :|
|: http://autobuild.org   -o- http://search.cpan.org/~danberr/ :|
|: http://entangle-photo.org   -o-   http://live.gnome.org/gtk-vnc :|
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[drago01]

On Mon, Jan 11, 2016 at 3:46 PM, Jan Kurik  wrote:
> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
>
> Change owner(s):
> * Florian Festi < ffesti AT redhat DOT com >
>
>
> Change format of the RPM Database from Berkeley DB to RPM's own format.
>
> == Detailed Description ==
> The current implementation of the RPM Database is based on Berkeley
> DB. There are doubts about the its future and level of maintenance. In
> addition rpm's use of the database has multiple issues on its own. As
> a result RPM upstream is working to replace the database format with a
> new implementation.
>
> == Scope ==
> Proposal owners:
> * Test and release the already implemented database format.
> * Make sure it is supported by all important tools
>   - Tools using librpm should not be affected
>   - Libsolv is the most prominent tool accessing the rpmdb directly
> and needs adjustment
> *Tooling for converting existing databases.

How will this work for upgrades? Is a db conversation needed before
the upgrade? Or can it be done afterwards?
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Neal Gompa]

On Mon, Jan 11, 2016 at 11:31 AM, Reindl Harald  wrote:
>
>
> Am 11.01.2016 um 17:26 schrieb Kalev Lember:
>>
>> On 01/11/2016 03:46 PM, Jan Kurik wrote:
>>>
>>> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
>>> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
>>>
>>> Change owner(s):
>>> * Florian Festi < ffesti AT redhat DOT com >
>>>
>>>
>>> Change format of the RPM Database from Berkeley DB to RPM's own format.
>>>
>>> == Detailed Description ==
>>> The current implementation of the RPM Database is based on Berkeley
>>> DB. There are doubts about the its future and level of maintenance. In
>>> addition rpm's use of the database has multiple issues on its own. As
>>> a result RPM upstream is working to replace the database format with a
>>> new implementation.
>>
>>
>> Is the new database going to be able to support yumdb use cases as well?
>> Might be a good time to get rid of separate rpmdb and yumdb and merge
>> them together.
>
>
> please don't do that
>
> the yumdb is bloat because it contains a endless history and that's growing
> on machines after 10, 20 or more dist-upgrades while currently it's easy to
> get rid of that bload by just rm -rf /var/lib/yum/* and/or rm -rf
> /var/lib/dnf/*
>

If they were merged, I suspect they'd add a command for cleaning up
history that you could use.



-- 
真実はいつも一つ！/ Always, there's only one truth!
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Kalev Lember]

On 01/11/2016 03:46 PM, Jan Kurik wrote:

= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
https://fedoraproject.org/wiki/Changes/NewRpmDBFormat

Change owner(s):
* Florian Festi < ffesti AT redhat DOT com >


Change format of the RPM Database from Berkeley DB to RPM's own format.

== Detailed Description ==
The current implementation of the RPM Database is based on Berkeley
DB. There are doubts about the its future and level of maintenance. In
addition rpm's use of the database has multiple issues on its own. As
a result RPM upstream is working to replace the database format with a
new implementation.


Is the new database going to be able to support yumdb use cases as well?
Might be a good time to get rid of separate rpmdb and yumdb and merge
them together.

Thanks,
Kalev
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Reindl Harald]

Am 11.01.2016 um 17:26 schrieb Kalev Lember:

On 01/11/2016 03:46 PM, Jan Kurik wrote:

= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
https://fedoraproject.org/wiki/Changes/NewRpmDBFormat

Change owner(s):
* Florian Festi < ffesti AT redhat DOT com >


Change format of the RPM Database from Berkeley DB to RPM's own format.

== Detailed Description ==
The current implementation of the RPM Database is based on Berkeley
DB. There are doubts about the its future and level of maintenance. In
addition rpm's use of the database has multiple issues on its own. As
a result RPM upstream is working to replace the database format with a
new implementation.


Is the new database going to be able to support yumdb use cases as well?
Might be a good time to get rid of separate rpmdb and yumdb and merge
them together.


please don't do that

the yumdb is bloat because it contains a endless history and that's 
growing on machines after 10, 20 or more dist-upgrades while currently 
it's easy to get rid of that bload by just rm -rf /var/lib/yum/* and/or 
rm -rf /var/lib/dnf/*




signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Reindl Harald]

Am 11.01.2016 um 17:33 schrieb Neal Gompa:

On Mon, Jan 11, 2016 at 11:31 AM, Reindl Harald  wrote:


Am 11.01.2016 um 17:26 schrieb Kalev Lember:


On 01/11/2016 03:46 PM, Jan Kurik wrote:


= Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
https://fedoraproject.org/wiki/Changes/NewRpmDBFormat

Change owner(s):
* Florian Festi < ffesti AT redhat DOT com >


Change format of the RPM Database from Berkeley DB to RPM's own format.

== Detailed Description ==
The current implementation of the RPM Database is based on Berkeley
DB. There are doubts about the its future and level of maintenance. In
addition rpm's use of the database has multiple issues on its own. As
a result RPM upstream is working to replace the database format with a
new implementation.



Is the new database going to be able to support yumdb use cases as well?
Might be a good time to get rid of separate rpmdb and yumdb and merge
them together.


please don't do that

the yumdb is bloat because it contains a endless history and that's growing
on machines after 10, 20 or more dist-upgrades while currently it's easy to
get rid of that bload by just rm -rf /var/lib/yum/* and/or rm -rf
/var/lib/dnf/*


If they were merged, I suspect they'd add a command for cleaning up
history that you could use


i also would have suspected that dnf and it's "lowlevel" libraries are 
using librpm and looking at that thread libsolv is accessing the 
database directly


as well i would have suspected that /var/lib/yum/ get some cleanup until 
i had enough by "locate fc12" find a ton of files with the history of 
all updates from intsrducing the yumdb on a F15 system


so i don't give anything about "suspect"





signature.asc
Description: OpenPGP digital signature
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard Hughes]

On 11 January 2016 at 14:46, Jan Kurik  wrote:
>   - Tools using librpm should not be affected

Is there any early-access to the new database e.g. some test rpms?
Quite a few of the PackageKit transactions have specific time requests
and I'd like to profile any new database for speed before the new
change is made. Thanks.

Richard.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Colin Walters]

On Mon, Jan 11, 2016, at 09:46 AM, Jan Kurik wrote:
> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat

It'd be interesting to know the technical details, worth reposting once there's
a design document or prototype PR.

I added rpm-ostree to the list, and one thing I'd like to note regarding that:
We always generate a new DB rather than mutate-in-place, because
rpm-ostree always generates atomic updates.  This means that
mutation speed is not very relevant to us, but query speed still
is.

It'd also be nice if the new format was cleanly mmap'able like
GVariant is.
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Dan Horák]

On Mon, 11 Jan 2016 15:46:27 +0100
Jan Kurik  wrote:

> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat
> 
> Change owner(s):
> * Florian Festi < ffesti AT redhat DOT com >
> 
> 
> Change format of the RPM Database from Berkeley DB to RPM's own
> format.
> 
> == Detailed Description ==
> The current implementation of the RPM Database is based on Berkeley
> DB. There are doubts about the its future and level of maintenance. In
> addition rpm's use of the database has multiple issues on its own. As
> a result RPM upstream is working to replace the database format with a
> new implementation.

does it mean rpm is changing from Berkeley DB library to another
library or to a completely new implementation of a database engine?


Dan
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org

Re: F24 System Wide Change: Change Proposal Name NewRpmDBFormat

[Richard W.M. Jones]

On Mon, Jan 11, 2016 at 03:46:27PM +0100, Jan Kurik wrote:
> = Proposed System Wide Change: Change Proposal Name NewRpmDBFormat =
> https://fedoraproject.org/wiki/Changes/NewRpmDBFormat

Details of the format?

What forward and backward compatibility guarantees are there?

libguestfs reads the rpmdb of guests (of any version, new or old) in
order to get the list of packages installed in a guest.  Remember that
the host and guest may be very different versions, eg. a RHEL 9 host
accessing a Fedora 24 guest.  Or vice versa.

Currently we use the (BDB-specific obviously) db_dump tool for this
purpose.  Other distros such as Debian as much simpler in this regard
since they expose the package data as plain text files.

https://github.com/libguestfs/libguestfs/blob/master/src/inspect-apps.c

Rich.

-- 
Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones
Read my programming and virtualization blog: http://rwmj.wordpress.com
libguestfs lets you edit virtual machines.  Supports shell scripting,
bindings from many languages.  http://libguestfs.org
--
devel mailing list
devel@lists.fedoraproject.org
http://lists.fedoraproject.org/admin/lists/devel@lists.fedoraproject.org