Bug#987013: Release goal proposal: Remove Berkeley DB
On Sat, 4 Feb 2023 08:50:29 +0100 Paul Gevers wrote: > > Remove Berkeley DB (finally) > > Sure. But I agree with several readers of this bug that there should > be a plan. We shouldn't kill it until the users are able to sanely > move away from it. I doubt that will happen automatically, so > somebody needs to organize it. Is there a reason against switching to this fork under the old license? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1010965 There are still applications using bdb by default and that means there should be tools to read and edit them for the next years to support a migration.
Bug#987013: Release goal proposal: Remove Berkeley DB
On Feb 04, Paul Gevers wrote: > I don't see the preparation happening in time for bookworm, so if the > preparations are done for trixie, Berkeley DB can be removed in forky. I object again to removing Berkeley DB: it is mature software and it works fine. At least inn2 uses it, and a "transition" (i.e. rebuilding the overview database with a different indexing method) for a non-trivial server may require hours of downtime. -- ciao, Marco signature.asc Description: PGP signature
Bug#987013: Release goal proposal: Remove Berkeley DB
Hi, As a Release Team member, I'm leave a small note here. On Thu, 15 Apr 2021 18:12:17 +0200 Bastian Blank wrote: I would like to propose a release goal: It has been a while since we did release goals in the formal way. I recommend instead to discuss this in a bigger audience and get traction amongst the Debian community. Remove Berkeley DB (finally) Sure. But I agree with several readers of this bug that there should be a plan. We shouldn't kill it until the users are able to sanely move away from it. I doubt that will happen automatically, so somebody needs to organize it. I don't see the preparation happening in time for bookworm, so if the preparations are done for trixie, Berkeley DB can be removed in forky. Paul OpenPGP_signature Description: OpenPGP digital signature
Bug#987013: Release goal proposal: Remove Berkeley DB
Apache httpd allows to use DBM file for various purposes. The default format is Berkeley DB. This is highly configuration dependent, automatic migration by maintainer scripts seems unfeasible. This means that the users need time and a tool to migrate their configurations. I have opened [1] for this. Subversion's first repository format uses Berkeley DB. But according to [2], new repositories have not used that format as default since 1.2 in 2005 and the format has been deprecated in 1.8 in 2013. So maybe a note in the release notes would be sufficient, here? [1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1030376 [2] https://en.wikipedia.org/wiki/Apache_Subversion
Bug#987013: Release goal proposal: Remove Berkeley DB
X-Debbugs-Cc: b...@debian.org Hi, cyrus-sasl2 package maintainer here. I am interested in the state of this since BerkeleyDB will be removed from sasl upstream with the next release [0]. There are several other implementations to choose from: gdbm|lmdb|ndbm. ndbm does not exist in Debian and gdbm's GPL license would probably be problematic when combined with CMU's BSD-4-clause code. Can someone shine a light on what a transition to lmdb would mean for cyrus-sasl2? Can the existing databases just be used as-is or do they need a migration? Cheers, Bastian [0]: https://github.com/cyrusimap/cyrus-sasl/issues/718
Bug#987013: Release goal proposal: Remove Berkeley DB
Matthias Klose wrote:
>> Then there's user code too. I also think we'll need at least a dumper
>> utility so that users can migrate their data manually when they discover
>> their program no longer works after upgrading.
>
> For Python, the dbm/ndbm.py module, based on the _dbm extension is also
> affected. You can build the _dbm extension using libgdbm-compat-dev, however
> that changes the on-disk format, and the license used (likely the new one
> should
> be moved into the python3-gdbm package).
Hi, I'm a nosy bystander.
Last year I was annoyed by scrapy using bdb to cache entire HTTP responses
(including large HTML bodies).
As an experiment, I wrote some proof-of-concept code for other backends.
IIRC if the database doesn't exist yet, they can drop-in replace "import dbm".
Here they are attached, do with them what you will.
I don't intend to touch them again myself.
(They are expat licensed, but I can relicense if needed.)
(FWIW, I eventually ended up patching scrapy to use sqlite3 directly, and then
gave up on scrapy entirely.)
# SYNOPSIS: a shim to use lmdb where in apps that expect dbm's API
#
# import dbm2lmdb as dbm
# db = dbm.open('porn')
# db['Alice'] = 'Bob'
# db['Alice']
# db['Clara']
# del db['Alice']
# db.close()
#
# SEE ALSO:
#
# https://en.wikipedia.org/wiki/Tokyo_Cabinet_and_Kyoto_Cabinet
# https://dbmx.net/kyotocabinet/pythondoc/ (seriously, frames?)
#
# NOTE: this is not even remotely working currently.
import pathlib
import logging
import kyotocabinet
class DBMLikeKyotoCabinet:
def __init__(self, path, flags=None, mode=None):
if flags is not None or mode is not None:
raise NotImplementedError(flags, mode)
# kyotocabinet databases MUST have a specific extension?
path_kch = pathlib.Path(path).with_suffix('.kch')
self.db = kyotocabinet.DB()
ok = self.db.open(path_kch)
if not ok: # seriously?
raise RuntimeError(self.db.error()) # seriously?
# db['foo']
def __getitem__(self, key):
value = self.db.get(_bytes(key))
if not value: # seriously?
logging.warn('%s', self.db.error) # seriously?
return None# seriously?
# db['foo'] = 'bar'
def __setitem__(self, key, value):
ok = self.db.set(_bytes(key), _bytes(value))
if not ok: # seriously?
raise RuntimeError(self.db.error()) # seriously?
# del db['foo']
def __delitem__(self, key):
raise NotImplementedError()
# 'foo' in db
# 'foo' not in db
def __contains__(self, key):
return self.__getitem__(key) is not None
def close(self):
return self.env.close()
def sync(self):
return self.env.sync()
def firstkey(self):
raise NotImplementedError()
def nextkey(self):
raise NotImplementedError()
def reorganize(self):
raise NotImplementedError()
def open(*args, **kwargs):
return DBMLikeKyotoCabinet(*args, **kwargs)
def whichdb(*args, **kwargs):
raise NotImplementedError()
def _bytes(b):
if isinstance(b, bytes):
return b
elif isinstance(b, str):
return bytes(b, encoding='UTF-8')
else:
raise ValueError(b)
# SYNOPSIS: a shim to use lmdb where in apps that expect dbm's API
#
# import dbm2lmdb as dbm
# db = dbm.open('porn')
# db['Alice'] = 'Bob'
# db['Alice']
# db['Clara']
# del db['Alice']
# db.close()
#
# SEE ALSO:
#
# https://lmdb.readthedocs.io
# http://www.lmdb.tech/doc/
import lmdb
class DBMLikeLMDBEnvironment:
def __init__(self,
path: str,
flags: str = 'r',
mode: int = 0o666):
for c in flags:
if c not in 'rwc':
raise NotImplementedError('Unsupported flag', c)
self.env = lmdb.Environment(
str(path), # str() to add pathlib support
readonly='r' in flags,
create='c' in flags)
# By default LMDB lets you store up to 10MiB; increase that to 1GiB.
# UPDATE: requires python3-lmdb (>= 0.87); Debian 10 has 0.86.
# Ouch! I give up for today.
self.env.set_mapsize(2**30)
# db['foo']
def __getitem__(self, key):
with self.env.begin() as txn:
return txn.get(_bytes(key))
# db['foo'] = 'bar'
def __setitem__(self, key, value):
with self.env.begin(write=True) as txn:
return txn.put(_bytes(key), _bytes(value))
# del db['foo']
def __delitem__(self, key):
with self.env.begin() as txn:
return txn.delete(_bytes(key))
# 'foo' in db
# 'foo' not in db
def __contains__(self, key):
return self.__getitem__(key) is not None
def close(self):
return self.env.close()
def sync(self):
return self.env.sync()
def firstkey(self):
Bug#987013: Release goal proposal: Remove Berkeley DB
On Fri, Apr 16, 2021 at 10:36:57PM +0200, Marco d'Itri wrote: > On Apr 16, Bastian Blank wrote: > > > postfix is easy. Would inn2 be license compliant with a AGPL licensed > > BDB, aka able to provide the source to it's users, or what is the plan > > anyway? > The plan is to continue using 5.3, not upgrading. > > > slapd defaults to LMDB since several years and you need to > > explicitely specify the bdb or hdb backend. > Sure, but the point was how to convert existing systems. As far as I can see, the realistic best case would be to drop Berkeley DB *after* bookworm. For usages that are not just build-time tests or temporary caches, we need at least one release for migrating the data of our users. apt-listchanges is using Berkeley DB through Python (#988090). This is one global database, and the user-friendly way of migration would be either in the maintainer scripts during the upgrade to bookworm or at runtime when the version in bookworm discovers a legacy Berkeley DB database. If Python in bookworm would not be able to read legacy Berkeley DB databases, we would be screwing our users by not being able to offer them automatic migrations in packages like apt-listchanges. I maintain bogofilter (a spam filter). It would be feasible to implement a transparent migration from Berkeley DB to a different format in bookworm, but this requires a bogofilter tool compiled against libdb5.3 in bookworm. Which would not be possible without libdb5.3 in bookworm. > ciao, > Marco cu Adrian
Bug#987013: Release goal proposal: Remove Berkeley DB
On 5/5/21 8:51 PM, Niko Tyni wrote: > On Fri, Apr 16, 2021 at 08:29:52PM +0200, Bastian Blank wrote: >> On Fri, Apr 16, 2021 at 05:04:03PM +0200, Marco d'Itri wrote: > >>> And then all the packages currently depending on libdb5.3 will need to >>> implement, or at least document, a transition strategy. >> >> My first goal would be to drop it from base packages, so not every >> system out there needs to have it installed. > > Hi, please note that there's a number of indirect users of libdb via > interpreter packages - at least Perl, presumably Python too. > > Given perl gets installed on almost all systems, this seems to to be > on the path to the first goal. > > For the perl core, the libdb5.3 bindings are exposed with the DB_File > module. I think this is the only place but have not cross-checked that. > (The libberkeleydb-perl package is an entirely separate matter AIUI.) > > I see 110 source packages in Debian matching DB_File. The list will > need to be inspected manually to weed out false positives. The remaining > packages need to be changed before perl can drop its libdb5.3 dependency. > I suppose this will also need a long list of Breaks declarations on the > perl side. > > Then there's user code too. I also think we'll need at least a dumper > utility so that users can migrate their data manually when they discover > their program no longer works after upgrading. For Python, the dbm/ndbm.py module, based on the _dbm extension is also affected. You can build the _dbm extension using libgdbm-compat-dev, however that changes the on-disk format, and the license used (likely the new one should be moved into the python3-gdbm package). Matthias
Bug#987013: Release goal proposal: Remove Berkeley DB
On Fri, Apr 16, 2021 at 08:29:52PM +0200, Bastian Blank wrote: > On Fri, Apr 16, 2021 at 05:04:03PM +0200, Marco d'Itri wrote: > > And then all the packages currently depending on libdb5.3 will need to > > implement, or at least document, a transition strategy. > > My first goal would be to drop it from base packages, so not every > system out there needs to have it installed. Hi, please note that there's a number of indirect users of libdb via interpreter packages - at least Perl, presumably Python too. Given perl gets installed on almost all systems, this seems to to be on the path to the first goal. For the perl core, the libdb5.3 bindings are exposed with the DB_File module. I think this is the only place but have not cross-checked that. (The libberkeleydb-perl package is an entirely separate matter AIUI.) I see 110 source packages in Debian matching DB_File. The list will need to be inspected manually to weed out false positives. The remaining packages need to be changed before perl can drop its libdb5.3 dependency. I suppose this will also need a long list of Breaks declarations on the perl side. Then there's user code too. I also think we'll need at least a dumper utility so that users can migrate their data manually when they discover their program no longer works after upgrading. -- Niko Tyni nt...@debian.org
Bug#987013: Release goal proposal: Remove Berkeley DB
Thanks for your explanation. Gerardo Il giorno ven 16 apr 2021 alle ore 20:09 Bastian Blank ha scritto: > > Hi Gerardo > > On Fri, Apr 16, 2021 at 10:30:08AM +0200, Gerardo Ballabio wrote: > > Bastian Blank wrote: > > > Berkeley DB was relicensed to AGPLv3 almost eight years ago. > > Sorry but I don't understand, why is that a problem? > > I believe the AGPL (you mean the GNU Affero General Public License, > > right?) is a free license. Is it not? > > Yes, the AGPLv3 is a free license. > > However the freeness is not the problem here. The problem is the AGPL, > it's extended source provisions, the incompatibility with the license of > existing software and also a bit "Oracle". > > The AGPL was created for network services. It requires to provide the > source to anyone accessing it via network. So this is tailored for the > services themselves, not arbitrary libraries deep within the dependency > chain. There where a lot of discussions about this problems at the > time.[1] > > So even if we would switch to a current version of Berkeley DB, we would > need to do the same work to make sure every software that uses it is in > compliance with the AGPL. AFAIK every distribution either stayed with > BDB 5.3 and often just removed it's use as much as possible or just > killed it alltogether. > > Regards, > Bastian > > [1]: https://lwn.net/Articles/557820/ > -- > If a man had a child who'd gone anti-social, killed perhaps, he'd still > tend to protect that child. > -- McCoy, "The Ultimate Computer", stardate 4731.3
Bug#987013: Release goal proposal: Remove Berkeley DB
On Apr 16, Bastian Blank wrote: > postfix is easy. Would inn2 be license compliant with a AGPL licensed > BDB, aka able to provide the source to it's users, or what is the plan > anyway? The plan is to continue using 5.3, not upgrading. > slapd defaults to LMDB since several years and you need to > explicitely specify the bdb or hdb backend. Sure, but the point was how to convert existing systems. -- ciao, Marco signature.asc Description: PGP signature
Bug#987013: Release goal proposal: Remove Berkeley DB
Hi Marco On Fri, Apr 16, 2021 at 05:04:03PM +0200, Marco d'Itri wrote: > On Apr 15, Bastian Blank wrote: > > After this time we really should try to get rid of this package, which > > even is NMU maintained since three years. > I am not persuaded. I maintain libberkeleydb-perl and it works fine, it > is mature software. Mature and unmaintained are not opposites. It works, but this does not mean it's advised to be used, especially if it works on untrusted data. (I the time I installed by current notebook, Linux 4.0 was the current version. It would still work, but would you really still use it on the internet?) > And then all the packages currently depending on libdb5.3 will need to > implement, or at least document, a transition strategy. My first goal would be to drop it from base packages, so not every system out there needs to have it installed. > Let me just mention postfix (easy), inn2 (possible but very resources > intensive) and slapd (I am not sure, but it is critical and scary). postfix is easy. Would inn2 be license compliant with a AGPL licensed BDB, aka able to provide the source to it's users, or what is the plan anyway? slapd defaults to LMDB since several years and you need to explicitely specify the bdb or hdb backend. Regards, Bastian -- Each kiss is as the first. -- Miramanee, Kirk's wife, "The Paradise Syndrome", stardate 4842.6
Bug#987013: Release goal proposal: Remove Berkeley DB
Hi Gerardo On Fri, Apr 16, 2021 at 10:30:08AM +0200, Gerardo Ballabio wrote: > Bastian Blank wrote: > > Berkeley DB was relicensed to AGPLv3 almost eight years ago. > Sorry but I don't understand, why is that a problem? > I believe the AGPL (you mean the GNU Affero General Public License, > right?) is a free license. Is it not? Yes, the AGPLv3 is a free license. However the freeness is not the problem here. The problem is the AGPL, it's extended source provisions, the incompatibility with the license of existing software and also a bit "Oracle". The AGPL was created for network services. It requires to provide the source to anyone accessing it via network. So this is tailored for the services themselves, not arbitrary libraries deep within the dependency chain. There where a lot of discussions about this problems at the time.[1] So even if we would switch to a current version of Berkeley DB, we would need to do the same work to make sure every software that uses it is in compliance with the AGPL. AFAIK every distribution either stayed with BDB 5.3 and often just removed it's use as much as possible or just killed it alltogether. Regards, Bastian [1]: https://lwn.net/Articles/557820/ -- If a man had a child who'd gone anti-social, killed perhaps, he'd still tend to protect that child. -- McCoy, "The Ultimate Computer", stardate 4731.3
Bug#987013: Release goal proposal: Remove Berkeley DB
On Apr 15, Bastian Blank wrote: > After this time we really should try to get rid of this package, which > even is NMU maintained since three years. I am not persuaded. I maintain libberkeleydb-perl and it works fine, it is mature software. But even if we agree that all the libdb5.3 reverse dependencies must migrate to a different database then probably we will need to keep around db5.3-util (and its dependency libdb5.3) to allow dumping and restoring the databases. Not all software uses libdb as a cache which can just be regenerated and/or supports multiple databases and has internal dump/restore tools. And then all the packages currently depending on libdb5.3 will need to implement, or at least document, a transition strategy. Let me just mention postfix (easy), inn2 (possible but very resources intensive) and slapd (I am not sure, but it is critical and scary). -- ciao, Marco signature.asc Description: PGP signature
Bug#987013: Release goal proposal: Remove Berkeley DB
Bastian Blank wrote: > Berkeley DB was relicensed to AGPLv3 almost eight years ago. Sorry but I don't understand, why is that a problem? I believe the AGPL (you mean the GNU Affero General Public License, right?) is a free license. Is it not? Gerardo
Bug#987013: Release goal proposal: Remove Berkeley DB
Hi, On 15.04.21 18:12, Bastian Blank wrote: > Package: release.debian.org > Severity: wishlist > > Hi > > I would like to propose a release goal: > > Remove Berkeley DB (finally) > > Berkeley DB was relicensed to AGPLv3 almost eight years ago. Since > then, Debian stayed with the last version before the license change. > The license change means, we can't take upstream patches, so security > support is only provided by other distributions with in the same state. > > After this time we really should try to get rid of this package, which > even is NMU maintained since three years. > > Affected source packages to remove: > - db-defaults > - db5.3 I second this proposal. Best regards, Martin -- Martin Zobel-Helas Technischer Leiter Betrieb Tel.: +49 2166 9901-0 Fax:+49 2166 9901-100 E-Mail: martin.zobel-he...@credativ.de pgp fingerprint: 6B18 5642 8E41 EC89 3D5D BDBB 53B1 AC6D B11B 627B http://www.credativ.de credativ GmbH, HRB Mönchengladbach 12080 USt-ID-Nummer: DE204566209 Trompeterallee 108, 41189 Mönchengladbach Geschäftsführung: Dr. Michael Meskes, Sascha Heuer Unser Umgang mit personenbezogenen Daten unterliegt folgenden Bestimmungen: https://www.credativ.de/datenschutz OpenPGP_signature Description: OpenPGP digital signature
Bug#987013: Release goal proposal: Remove Berkeley DB
Package: release.debian.org Severity: wishlist Hi I would like to propose a release goal: Remove Berkeley DB (finally) Berkeley DB was relicensed to AGPLv3 almost eight years ago. Since then, Debian stayed with the last version before the license change. The license change means, we can't take upstream patches, so security support is only provided by other distributions with in the same state. After this time we really should try to get rid of this package, which even is NMU maintained since three years. Affected source packages to remove: - db-defaults - db5.3 Bastian
