Your message dated Sat, 28 Aug 2021 18:03:48 +0000 with message-id <[email protected]> and subject line Bug#983914: fixed in gdbm 1.20-1 has caused the Debian Bug report #983914, regarding libgdbm6: there is file descriptor leak in recover.c when using mmaped file to be marked as done.
This means that you claim that the problem has been dealt with. If this is not the case it is now your responsibility to reopen the Bug report if necessary, and/or fix the problem forthwith. (NB: If you are a system administrator and have no idea what this message is talking about, this may indicate a serious mail system misconfiguration somewhere. Please contact [email protected] immediately.) -- 983914: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=983914 Debian Bug Tracking System Contact [email protected] with problems
--- Begin Message ---Package: libgdbm6 Version: 1.18.1-4 Severity: normal Tags: upstream patch Dear Maintainer, We are using gdbm through python, and discovered that on debian buster our project started to leak file descriptors. After investigating the issue we came to the conclusion that it is caused by a bug in the libgdbm6 library. For reproduction on debian:buster docker image run the following commands: apt install -y python3 python3-gdbm lsof cat >/tmp/repro.py <<EOL #!/usr/bin/python3 import dbm.gnu # IMPORTANT import os import time import subprocess def main(): db_file = "/tmp/ti-db-hanging.db" # call reorganize() multiple times for _ in range(5): db = dbm.open(os.path.abspath(db_file), "cf") db.reorganize() time.sleep(1) db.close() time.sleep(5) # check if we still hold any file handles for the db_file even after closing it pid = os.getpid() x = subprocess.run([f"lsof -p {pid} | grep {db_file}"], shell=True, stdout=subprocess.PIPE) assert x.returncode == 0 res = x.stdout.decode() if not res: print("check open files: ok") else: print(res) if __name__ == "__main__": main() EOL python3 /tmp/repro.py In the end of the script lsof should not return any open file handles but it does show 5 of them. Short summary of the issue: When gdbm reorganize is called from python it will end up calling the gdbm_recover in recover.c . This will create a new temporary file where the contents of the current file will be copied (after doing some processing on it). After this is done new file will renamed to the old one. When mmap is used the mapping for the temporary file is only removed in case of error situations. This results in file handles hold on DELETED files when the process runs reorganize more than once in its lifetime. To fix the issue I prepared the following patch on gdbm: In case mmap is used, the memory mapping in the recover function is not removed. This can cause open file descriptors to deleted files if the recover function is called multiple times. --- a/src/recover.c +++ b/src/recover.c @@ -168,15 +168,20 @@ dbf->bucket_changed = new_dbf->bucket_changed; dbf->second_changed = new_dbf->second_changed; - free (new_dbf->name); - free (new_dbf); - #if HAVE_MMAP /* Re-initialize mapping if required */ if (dbf->memory_mapping) _gdbm_mapped_init (dbf); + + /* remove the old memory mapping to the temporary file name */ + if (new_dbf->mapped_region){ + _gdbm_mapped_unmap(new_dbf); + } #endif + free (new_dbf->name); + free (new_dbf); + /* Make sure the new database is all on disk. */ gdbm_file_sync (dbf); -- System Information: Debian Release: 10.7 APT prefers stable-updates APT policy: (500, 'stable-updates'), (500, 'stable') Architecture: amd64 (x86_64) Kernel: Linux 5.4.0-66-generic (SMP w/4 CPU cores) Locale: LANG=C, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=C (charmap=UTF-8) Shell: /bin/sh linked to /bin/dash Init: unable to detect Versions of packages libgdbm6 depends on: ii libc6 2.28-10 libgdbm6 recommends no packages. Versions of packages libgdbm6 suggests: pn gdbm-l10n <none> -- no debconf information
--- End Message ---
--- Begin Message ---Source: gdbm Source-Version: 1.20-1 Done: Nicolas Mora <[email protected]> We believe that the bug you reported is fixed in the latest version of gdbm, which is due to be installed in the Debian FTP archive. A summary of the changes between this version and the previous one is attached. Thank you for reporting the bug, which will now be closed. If you have further comments please address them to [email protected], and the maintainer will reopen the bug report if appropriate. Debian distribution maintenance software pp. Nicolas Mora <[email protected]> (supplier of updated gdbm package) (This message was generated automatically at their request; if you believe that there is a problem with it please contact the archive administrators by mailing [email protected]) -----BEGIN PGP SIGNED MESSAGE----- Hash: SHA512 Format: 1.8 Date: Fri, 16 Jul 2021 09:05:53 -0400 Source: gdbm Architecture: source Version: 1.20-1 Distribution: unstable Urgency: medium Maintainer: Nicolas Mora <[email protected]> Changed-By: Nicolas Mora <[email protected]> Closes: 983914 991028 Changes: gdbm (1.20-1) unstable; urgency=medium . * New maintainer upload (Closes: #991028) * New upstream version 1.20 * Update Standards-Version to 4.5.1 * d/control: Update debhelper version to 13 * d/upstream/metadata: Add Repository and Repository-Browse * d/patches/recover.patch: Fix file descriptor leak (Closes: #983914) * libgdbm-compat-dev: Install libgdbm_compat.a Checksums-Sha1: 8670c200432bfb34eed3a50117a439a8cd2e0a7e 2604 gdbm_1.20-1.dsc f3200c9c4e5c7de4978f838bbabeaa7c5b857bd7 987628 gdbm_1.20.orig.tar.gz 4ed60e1bd9202b9ad12cd7a90ef269820cdf6139 181 gdbm_1.20.orig.tar.gz.asc e892359870604b0afe94859a5b8445e4b075a88d 16800 gdbm_1.20-1.debian.tar.xz a098ada6b0928c58e2ef42b90602dd4f5d847be1 8567 gdbm_1.20-1_amd64.buildinfo Checksums-Sha256: d46106cfac55f94cf0bc49302800cdbd38aada518bf61cb032b07d255f0753c3 2604 gdbm_1.20-1.dsc 3aeac05648b3482a10a2da986b9f3a380a29ad650be80b9817a435fb8114a292 987628 gdbm_1.20.orig.tar.gz f0fed968dd6b841109d9a1bc44242522bc134af74bdf3473345d33bb19dc67af 181 gdbm_1.20.orig.tar.gz.asc 5eb6272edf4184707e480716d1f39ad449ea6125f41bf194e9c4b29c94a1462b 16800 gdbm_1.20-1.debian.tar.xz 9836afac5a76c8112c08065463c5ab4a8be105c162443866a7ab6ebbc11f8b1a 8567 gdbm_1.20-1_amd64.buildinfo Files: 1b13b7df84f1fa7a16892d6ade5092a7 2604 libs optional gdbm_1.20-1.dsc 006c19b8b60828fd6916a16f3496bd3c 987628 libs optional gdbm_1.20.orig.tar.gz 5bf2d1b2fe07a8defa9546f39cf36907 181 libs optional gdbm_1.20.orig.tar.gz.asc 4f88d715d22f75d49960015b61b7a28f 16800 libs optional gdbm_1.20-1.debian.tar.xz 8db7e8c731a6e9dfaee7d3fd196fa53d 8567 libs optional gdbm_1.20-1_amd64.buildinfo -----BEGIN PGP SIGNATURE----- iQIzBAEBCgAdFiEEhAWwL8wo75dEyPJT/oITlEC9IrkFAmEqcgoACgkQ/oITlEC9 IrmqtRAAjcUpveLT2aqPvr3OWfncY+l0iclV0ANA4HgyZ+r1lTqqPOMzmMO14IAV FMKvNgMGcvg+wKrcP3jEj++nD6K2NcdfXY1Wm1oX29+p5SqVBKQNB8CHfr8BF+Yl LBXETNFORGOtZZWychW+4px56kiv5l0TB2FogcZb3N1K7mQcwLZd7a/TCAbSCH/l 6TPWDbynO5Y+YmDij4yvmSh9bwK46XSsfqX2HOuUljhq1YBbb3RPGT42yDYM3u5k BcJ4ssLNVhYWRBVtYctvSEQ82+jDOhTtzC+XZN4vLuKoEa1YnmjIyZy6j8jTVYhW RGgkEkk0MNy0oVVQs7lfEwGBonsJfgpbZFyqXDoCD1YilKYx5zG+zZYLF+JSmnlz 8If6ye0ZtPdnWdsxVfjYg+IFQ5rfGLPKiBXDy2scnyMceNvNX1NoX7nz7LVFDP0n 5Y67Mmg/bNy6lvFAQc92+GyvT8WVgCKWvBXuvfgnRUme2X88QebBU2cEKtaCPWlQ 51uN6GeaY5TNsXijch8zTmp0El18ZMk/J550PPpHTwKt7vvzGTkWsfja8NU+b+35 nwsPRDEmxBiQOUTsncTqr0pwRY0ZpYkDXyUf7BJFpOdT+oDukuq8rVXwHK7msHG1 ZY5Yb1bJfTMf6NnKMTOdZccK90cRwYxO6mU6PhGrtAqBRvs/SF0= =42bZ -----END PGP SIGNATURE-----
--- End Message ---
