On Sat, Aug 09, 2025 at 02:30:34PM -0400, Matt Corallo wrote: > I tend to use bash_history as a way to remember specific commands to run for > various tasks, often quite irregularly, and thus keep HISTFILESIZE quite > large (800k lines in my .bash_history, for 32MiB).
Not a real solution but maybe history deduplication (which can be nice for other reasons) would reduce the need for a large history. > bash/readline handle this > just fine, except that I often have something like 50 instances of bash > running in various terminal emulators. > This leads to something like 7.5GB of > memory dedicated to bash instances (around 150MB per instance), with most of > it going to history storage. > > While various optimizations to improve the space efficiency of history would > be welcome, ultimately this really feels like a task for Linux's samepage > merging logic. If the history strings themselves were put into contiguous > memory (okay, not incredibly trivial, but arenas aren't so complicated), it > should be fairly straightforward to just `madvise(addr, length, > MADV_MERGEABLE)` and let KSM handle the rest. Interesting, this does seem like a possible solution. (Instead of parsing history up-front, one could also store the entire ~/.bash_history in memory verbatim, and decode it whenever accessing history. Not saying that this is more elegant than the arena approach, but it's probably an option) > I can of course set a lower HISTSIZE than HISTFILESIZE and set histappend to > only load some of history in memory, but even the last 10k lines of history > leaves me often grep'ing bash_history manually. Maybe I'm missing some > obvious way to reduce redundant many-bash-instance memory usage, though? fish shell has a hardcoded maximum of 256k (deduplicated) history items, and gets away with it by using mmap() -- except on remote file systems, where it assumes that mmap() doesn't work (well). See https://github.com/fish-shell/fish-shell/blob/894d4ecc/src/history.rs#L1-L15 It might work for readline too, though of course it would make "sed 1d ~/.bash_history" crash bash ("Memory access within the mapping but beyond the current end of the underlying objects may result in SIGBUS signals being sent to the process." and "It is unspecified whether modifications to the underlying object done after the MAP_PRIVATE mapping is established are visible through the MAP_PRIVATE mapping.") fish assumes that nothing like "sed 1d ..." will happen; all non-appending edits to the file are done via copy-and-rename-into-place, which creates a new inode. > > Thanks, > Matt >
