Package: release.debian.org Severity: normal Tags: trixie X-Debbugs-Cc: [email protected] Control: affects -1 + src:ust User: [email protected] Usertags: pu
[ Reason ] Using an application instrumented with lttng-ust in an environment where RLIMIT_NOFILE is set to a high value like 1073741816 results in unexpected memory usage. This behavior as been fixed upstream but is being reported by users of the trixie package as it affects any program linked against liblttng-ust1t64. The debian bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1148883 The upstream bug: https://bugs.lttng.org/issues/1438 [ Impact ] Unexpected memory usage for programs linked with liblttng-ust1t64. This has been reported by users in docker containers. [ Tests ] Tested in a trixie VM with the upstream LTTng-tools test suite. [ Risks ] This is a cherry-pick of the patch backported in the upstream project stable updates, the risk of regression is pretty low. [ Checklist ] [x] *all* changes are documented in the d/changelog [x] I reviewed all changes and I approve them [x] attach debdiff against the package in (old)stable [x] the issue is verified as fixed in unstable [ Changes ] Add an upstream patch that replaces an array of RLIMIT_NOFILE file descriptors by a hash table that only contains the allocated file descriptors of the running program.
diff -Nru ust-2.13.9/debian/changelog ust-2.13.9/debian/changelog --- ust-2.13.9/debian/changelog 2025-04-14 17:01:33.000000000 -0400 +++ ust-2.13.9/debian/changelog 2026-11-11 09:01:43.000000000 -0500 @@ -1,3 +1,10 @@ +ust (2.13.9-1+deb13u1) trixie; urgency=medium + + * Set gbp config for trixie branch + * Fix excessive memory usage with large RLIMIT_NOFILE (Closes: #1148883) + + -- Michael Jeanson <[email protected]> Wed, 11 Nov 2026 09:01:43 -0500 + ust (2.13.9-1) unstable; urgency=medium * [e373746] New upstream version 2.13.9 diff -Nru ust-2.13.9/debian/gbp.conf ust-2.13.9/debian/gbp.conf --- ust-2.13.9/debian/gbp.conf 2024-05-24 15:17:56.000000000 -0400 +++ ust-2.13.9/debian/gbp.conf 2026-11-11 08:57:48.000000000 -0500 @@ -1,3 +1,4 @@ [DEFAULT] -upstream-branch=upstream/latest -debian-branch=debian/sid +pristine-tar=True +upstream-branch=upstream/trixie +debian-branch=debian/trixie diff -Nru ust-2.13.9/debian/patches/0001-fix-excessive-memory-usage-of-fd-tracker.patch ust-2.13.9/debian/patches/0001-fix-excessive-memory-usage-of-fd-tracker.patch --- ust-2.13.9/debian/patches/0001-fix-excessive-memory-usage-of-fd-tracker.patch 1969-12-31 19:00:00.000000000 -0500 +++ ust-2.13.9/debian/patches/0001-fix-excessive-memory-usage-of-fd-tracker.patch 2026-11-11 08:58:28.000000000 -0500 @@ -0,0 +1,391 @@ +From 70b3920c7acd37def7a04d3946c803f09a802565 Mon Sep 17 00:00:00 2001 +From: Olivier Dion <[email protected]> +Date: Wed, 1 Apr 2026 15:11:11 -0400 +Subject: [PATCH 1/1] fix: excessive memory usage of fd-tracker + +Issue +===== + +The fd tracker allocates an array of fd_set structures sized according +to the RLIMIT_NOFILE hard limit. On systems where this limit is very +high (e.g., Docker containers with systemd v240+ where fs.nr_open can be +1,073,741,816), this results in ~128 MB of memory allocation per +process. + +Solution +======== + +Replace the fd_set array with a hash table using cds_hlist from +liburcu. The hash table uses 1024 fixed buckets and dynamically +allocates entries as more file descriptors are tracked. + +Memory comparison for typical usage (10 tracked fds): + - Before: ~128 MB + - After: ~8 KB (fixed) + ~240 bytes (10 entries) + +The hash table provides O(1) average-case lookup, insertion, and +deletion, matching the performance characteristics of the previous +fd_set implementation. + +Known drawbacks +=============== + +The operations in the "close range" family can be slower in certain +configuration due to lookups in the hash table. + +Fix: https://bugs.lttng.org/issues/1438 +Signed-off-by: Olivier Dion <[email protected]> +Signed-off-by: Mathieu Desnoyers <[email protected]> +Change-Id: I16c7b48797e25a72737da9c14d2b702f96e75767 +Signed-off-by: Felix Moessbauer <[email protected]> +--- + src/lib/lttng-ust-common/fd-tracker.c | 197 ++++++++++++++++---------- + 1 file changed, 123 insertions(+), 74 deletions(-) + +diff --git a/src/lib/lttng-ust-common/fd-tracker.c b/src/lib/lttng-ust-common/fd-tracker.c +index 102e5c05..88d94889 100644 +--- a/src/lib/lttng-ust-common/fd-tracker.c ++++ b/src/lib/lttng-ust-common/fd-tracker.c +@@ -14,7 +14,6 @@ + #include <assert.h> + #include <errno.h> + #include <fcntl.h> +-#include <sys/select.h> + #include <sys/resource.h> + #include <sys/time.h> + #include <fcntl.h> +@@ -22,6 +21,7 @@ + #include <signal.h> + #include <stdbool.h> + #include <urcu/compiler.h> ++#include <urcu/hlist.h> + #include <urcu/tls-compat.h> + #include <urcu/system.h> + +@@ -33,24 +33,21 @@ + + #include "lib/lttng-ust-common/fd-tracker.h" + +-/* Operations on the fd set. */ +-#define IS_FD_VALID(fd) ((fd) >= 0 && (fd) < lttng_ust_max_fd) +-#define GET_FD_SET_FOR_FD(fd, fd_sets) (&((fd_sets)[(fd) / FD_SETSIZE])) +-#define CALC_INDEX_TO_SET(fd) ((fd) % FD_SETSIZE) +-#define IS_FD_STD(fd) (IS_FD_VALID(fd) && (fd) <= STDERR_FILENO) ++/* Hash table for tracking fds used by lttng-ust. */ ++#define FD_TRACKER_HT_BITS 10U ++#define FD_TRACKER_HT_SIZE (1U << FD_TRACKER_HT_BITS) + +-/* Check fd validity before calling these. */ +-#define ADD_FD_TO_SET(fd, fd_sets) \ +- FD_SET(CALC_INDEX_TO_SET(fd), GET_FD_SET_FOR_FD(fd, fd_sets)) +-#define IS_FD_SET(fd, fd_sets) \ +- FD_ISSET(CALC_INDEX_TO_SET(fd), GET_FD_SET_FOR_FD(fd, fd_sets)) +-#define DEL_FD_FROM_SET(fd, fd_sets) \ +- FD_CLR(CALC_INDEX_TO_SET(fd), GET_FD_SET_FOR_FD(fd, fd_sets)) ++#define IS_FD_STD(fd) ((fd) >= 0 && (fd) <= STDERR_FILENO) ++ ++struct lttng_fd_entry { ++ struct cds_hlist_node hlist; ++ int fd; ++}; + + /* +- * Protect the lttng_fd_set. Nests within the ust_lock, and therefore +- * within the libc dl lock. Therefore, we need to allocate the TLS before +- * nesting into this lock. ++ * Protect fd_tracker_ht. Nests within the ust_lock, and therefore within the ++ * libc dl lock. Therefore, we need to allocate the TLS before nesting into this ++ * lock. + * + * The ust_safe_guard_fd_mutex nests within the ust_mutex. This mutex + * is also held across fork. +@@ -65,12 +62,40 @@ static pthread_mutex_t ust_safe_guard_fd_mutex = PTHREAD_MUTEX_INITIALIZER; + */ + static DEFINE_URCU_TLS(int, ust_fd_mutex_nest); + +-/* fd_set used to book keep fd being used by lttng-ust. */ +-static fd_set *lttng_fd_set; +-static int lttng_ust_max_fd; +-static int num_fd_sets; ++/* Hash table used to book keep fd being used by lttng-ust. */ ++static struct cds_hlist_head fd_tracker_ht[FD_TRACKER_HT_SIZE]; ++static unsigned int lttng_ust_max_fd; + static int init_done; + ++static inline struct cds_hlist_head *fd_tracker_get_bucket(int fd) ++{ ++ return &fd_tracker_ht[(unsigned int) fd % FD_TRACKER_HT_SIZE]; ++} ++ ++static struct lttng_fd_entry *fd_tracker_lookup(int fd) ++{ ++ const struct cds_hlist_head *head; ++ const struct cds_hlist_node *node; ++ struct lttng_fd_entry *entry; ++ ++ if (caa_unlikely(fd < 0)) ++ return NULL; ++ ++ head = fd_tracker_get_bucket(fd); ++ ++ cds_hlist_for_each_entry(entry, node, head, hlist) { ++ if (entry->fd == fd) ++ return entry; ++ } ++ ++ return NULL; ++} ++ ++static inline bool fd_tracker_is_fd_tracked(int fd) ++{ ++ return fd_tracker_lookup(fd) != NULL; ++} ++ + /* + * Force a read (imply TLS allocation for dlopen) of TLS variables. + */ +@@ -80,41 +105,43 @@ void lttng_ust_fd_tracker_alloc_tls(void) + } + + /* +- * Allocate the fd set array based on the hard limit set for this +- * process. This will be called during the constructor execution ++ * Initialize the fd tracker hash table. ++ * This will be called during the constructor execution + * and will also be called in the child after fork via lttng_ust_init. + */ + void lttng_ust_fd_tracker_init(void) + { +- struct rlimit rlim; +- int i; ++ size_t i; + + if (CMM_LOAD_SHARED(init_done)) + return; + +- memset(&rlim, 0, sizeof(rlim)); +- /* Get the current possible max number of fd for this process. */ +- if (getrlimit(RLIMIT_NOFILE, &rlim) < 0) +- abort(); +- /* +- * FD set array size determined using the hard limit. Even if +- * the process wishes to increase its limit using setrlimit, it +- * can only do so with the softlimit which will be less than the +- * hard limit. +- */ +- lttng_ust_max_fd = rlim.rlim_max; +- num_fd_sets = lttng_ust_max_fd / FD_SETSIZE; +- if (lttng_ust_max_fd % FD_SETSIZE) +- ++num_fd_sets; +- if (lttng_fd_set != NULL) { +- free(lttng_fd_set); +- lttng_fd_set = NULL; ++ ++ if (!lttng_ust_max_fd) { ++ struct rlimit rlim; ++ ++ if (getrlimit(RLIMIT_NOFILE, &rlim) < 0) ++ abort(); ++ ++ lttng_ust_max_fd = rlim.rlim_max; ++ } ++ ++ for (i = 0; i < FD_TRACKER_HT_SIZE; ++i) { ++ struct cds_hlist_head *head; ++ struct cds_hlist_node *node, *tmp; ++ struct lttng_fd_entry *entry; ++ ++ head = &fd_tracker_ht[i]; ++ ++ /* Clear any existing entries (can happen after fork). */ ++ cds_hlist_for_each_entry_safe(entry, node, tmp, head, hlist) { ++ cds_hlist_del(&entry->hlist); ++ free(entry); ++ } ++ ++ CDS_INIT_HLIST_HEAD(head); + } +- lttng_fd_set = malloc(num_fd_sets * (sizeof(fd_set))); +- if (!lttng_fd_set) +- abort(); +- for (i = 0; i < num_fd_sets; i++) +- FD_ZERO((<tng_fd_set[i])); ++ + CMM_STORE_SHARED(init_done, 1); + } + +@@ -238,6 +265,14 @@ error: + return ret; + } + ++static void fd_tracker_add(struct lttng_fd_entry *entry) ++{ ++ struct cds_hlist_head *head; ++ ++ head = fd_tracker_get_bucket(entry->fd); ++ cds_hlist_add_head(&entry->hlist, head); ++} ++ + /* + * Needs to be called with ust_safe_guard_fd_mutex held when opening the fd. + * Has strict checking of fd validity. +@@ -253,7 +288,11 @@ error: + */ + int lttng_ust_add_fd_to_tracker(int fd) + { +- int ret; ++ struct lttng_fd_entry *entry = NULL; ++ ++ if (caa_unlikely(fd < 0)) ++ return -EBADF; ++ + /* + * Ensure the tracker is initialized when called from + * constructors. +@@ -261,23 +300,32 @@ int lttng_ust_add_fd_to_tracker(int fd) + lttng_ust_fd_tracker_init(); + assert(URCU_TLS(ust_fd_mutex_nest)); + ++ entry = zmalloc(sizeof(*entry)); ++ if (!entry) ++ return -ENOMEM; ++ + if (IS_FD_STD(fd)) { +- ret = dup_std_fd(fd); +- if (ret < 0) { +- goto error; ++ fd = dup_std_fd(fd); ++ if (fd < 0) { ++ free(entry); ++ goto end; + } +- fd = ret; + } + +- /* Trying to add an fd which we can not accommodate. */ +- assert(IS_FD_VALID(fd)); +- /* Setting an fd thats already set. */ +- assert(!IS_FD_SET(fd, lttng_fd_set)); ++ /* Setting an fd that's already set. */ ++ assert(!fd_tracker_is_fd_tracked(fd)); + +- ADD_FD_TO_SET(fd, lttng_fd_set); ++ entry->fd = fd; ++ ++ fd_tracker_add(entry); ++end: + return fd; +-error: +- return ret; ++} ++ ++static void fd_tracker_delete(struct lttng_fd_entry *entry) ++{ ++ cds_hlist_del(&entry->hlist); ++ free(entry); + } + + /* +@@ -286,6 +334,8 @@ error: + */ + void lttng_ust_delete_fd_from_tracker(int fd) + { ++ struct lttng_fd_entry *entry; ++ + /* + * Ensure the tracker is initialized when called from + * constructors. +@@ -293,12 +343,11 @@ void lttng_ust_delete_fd_from_tracker(int fd) + lttng_ust_fd_tracker_init(); + + assert(URCU_TLS(ust_fd_mutex_nest)); +- /* Not a valid fd. */ +- assert(IS_FD_VALID(fd)); + /* Deleting an fd which was not set. */ +- assert(IS_FD_SET(fd, lttng_fd_set)); ++ entry = fd_tracker_lookup(fd); ++ assert(entry); + +- DEL_FD_FROM_SET(fd, lttng_fd_set); ++ fd_tracker_delete(entry); + } + + /* +@@ -326,7 +375,7 @@ int lttng_ust_safe_close_fd(int fd, int (*close_cb)(int fd)) + return close_cb(fd); + + lttng_ust_lock_fd_tracker(); +- if (IS_FD_VALID(fd) && IS_FD_SET(fd, lttng_fd_set)) { ++ if (fd_tracker_is_fd_tracked(fd)) { + ret = -1; + errno = EBADF; + } else { +@@ -364,7 +413,7 @@ int lttng_ust_safe_fclose_stream(FILE *stream, int (*fclose_cb)(FILE *stream)) + fd = fileno(stream); + + lttng_ust_lock_fd_tracker(); +- if (IS_FD_VALID(fd) && IS_FD_SET(fd, lttng_fd_set)) { ++ if (fd_tracker_is_fd_tracked(fd)) { + ret = -1; + errno = EBADF; + } else { +@@ -399,7 +448,7 @@ static int test_close_success(const int *p __attribute__((unused))) + */ + int lttng_ust_safe_closefrom_fd(int lowfd, int (*close_cb)(int fd)) + { +- int ret = 0, close_success = 0, i; ++ int ret = 0, close_success = 0; + + lttng_ust_fd_tracker_alloc_tls(); + +@@ -422,7 +471,7 @@ int lttng_ust_safe_closefrom_fd(int lowfd, int (*close_cb)(int fd)) + * validating whether the FD is part of the tracked set. + */ + if (URCU_TLS(ust_fd_mutex_nest)) { +- for (i = lowfd; i < lttng_ust_max_fd; i++) { ++ for (unsigned int i = lowfd; i < lttng_ust_max_fd; i++) { + if (close_cb(i) < 0) { + switch (errno) { + case EBADF: +@@ -437,8 +486,8 @@ int lttng_ust_safe_closefrom_fd(int lowfd, int (*close_cb)(int fd)) + } + } else { + lttng_ust_lock_fd_tracker(); +- for (i = lowfd; i < lttng_ust_max_fd; i++) { +- if (IS_FD_VALID(i) && IS_FD_SET(i, lttng_fd_set)) ++ for (unsigned int i = lowfd; i < lttng_ust_max_fd; i++) { ++ if (fd_tracker_is_fd_tracked(i)) + continue; + if (close_cb(i) < 0) { + switch (errno) { +@@ -473,7 +522,7 @@ end: + int lttng_ust_safe_close_range_fd(unsigned int first, unsigned int last, int flags, + int (*close_range_cb)(unsigned int first, unsigned int last, int flags)) + { +- int ret = 0, i; ++ int ret = 0; + + lttng_ust_fd_tracker_alloc_tls(); + +@@ -498,13 +547,13 @@ int lttng_ust_safe_close_range_fd(unsigned int first, unsigned int last, int fla + goto end; + } + } else { +- int last_check = last; ++ unsigned int last_check = min_t(unsigned int, ++ last, ++ lttng_ust_max_fd); + +- if (last > lttng_ust_max_fd) +- last_check = lttng_ust_max_fd; + lttng_ust_lock_fd_tracker(); +- for (i = first; i <= last_check; i++) { +- if (IS_FD_VALID(i) && IS_FD_SET(i, lttng_fd_set)) ++ for (unsigned int i = first; i <= last_check; i++) { ++ if (fd_tracker_is_fd_tracked(i)) + continue; + if (close_range_cb(i, i, flags) < 0) { + ret = -1; +-- +2.55.0 + diff -Nru ust-2.13.9/debian/patches/series ust-2.13.9/debian/patches/series --- ust-2.13.9/debian/patches/series 2024-05-24 10:47:53.000000000 -0400 +++ ust-2.13.9/debian/patches/series 2026-11-11 08:59:32.000000000 -0500 @@ -1,2 +1,3 @@ fix-jni-libdir.patch use-python3.patch +0001-fix-excessive-memory-usage-of-fd-tracker.patch
