Re: Memory allocation failed during fsck of large EXT4 filesystem
Hello, On Tue, Jul 06, 2021 at 02:34:30PM +0300, IL Ka wrote: > > I use a 32bit OS Is the hardware capable of 64-bit? If so then it should be possible to install an amd64 kernel and e2fsprogs without completely converting your system to amd64. https://wiki.debian.org/CrossGrading (Stop after booting in to the new kernel and then install e2fsprogs:amd64) You should then be able to fsck larger ext* filesystems. There is also the "crossgrader": https://packages.debian.org/bullseye/crossgrader (though it is only in testing and unstable it is intended to work on stable as well) You may or may not find that simpler. Cheers, Andy -- https://bitfolk.com/ -- No-nonsense VPS hosting
Re: Memory allocation failed during fsck of large EXT4 filesystem
> I use a 32bit OS > > 32-bit OS can't use more than 4GB. 32-bit app can use even lower amount of memory. This is why 50GB swap file didn't help.
Re: Memory allocation failed during fsck of large EXT4 filesystem
Hi, Reiner Buehl wrote. > I would like to fix the filesystem > so that I can then use more intelligent recovery methods that do not need to > copy every file. Maybe the old workaround proposed by Ted T'so in https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=614082 would be worth a try. Bug reporter: "Error storing directory block information (inode=169246423, block=0, num=3966024): Memory allocation failed" Ted T'so: "The way to work around this problem should it occur again is to note the inode number, and then zap it using debugfs: debugfs -w /dev/md1 debugfs: clri <169246423> debugfs: quit Then e2fsck should be able to complete its work." If possible i would make any experiments on a plain copy of the filesystem. I.e. a copy of the bytes which can be read from the partition device. - Far fetched approach: If this would not help and i would deem it worth the effort, then i'd get the newest source of e2fsprogs and try to build it. Then i'd run the new fsck.ext4 program on a copy of the filesystem. If it still fails with "Memory allocation failed" then i'd try to let the program print a message from any occasion of EXT2_ET_NO_MEMORY, describing which occasion was triggered. This point in the source and, if possible, a stack trace would be hands-on information for the developers of e2fsprogs. I'm not sure how good an idea it is to run fsck under gdb to get a stack trace. There are glib functions backtrace(3) and backtrace_symbols_fd(3). https://www.gnu.org/software/libc/manual/html_node/Backtraces.html Their usage is shown in the handler function of https://stackoverflow.com/questions/77005/how-to-automatically-generate-a-stacktrace-when-my-program-crashes In your case no signal would be received, of course. You'd call the trace printer from the various EXT2_ET_NO_MEMORY occasions. As said: far fetched and open ended ... Have a nice day :) Thomas
Re: Memory allocation failed during fsck of large EXT4 filesystem
/ and /home are fine on the system. The data on the affected filesystem is a collection of data from different remote sites, so it could be restored but that would take a lot of time. That's why I would like to fix the filesystem so that I can then use more intelligent recovery methods that do not need to copy every file. Am Mo., 5. Juli 2021 um 23:21 Uhr schrieb Thomas D. Dean < tomd...@wavecable.com>: > On 7/5/21 1:54 PM, Michael Stone wrote: > > On Mon, Jul 05, 2021 at 12:53:39PM +0300, IL Ka wrote: > >> 7TB seems like too much for one partition imho. > >> Consider splitting it into the parts > > > > That's silly. It's 2021; 7TB isn't particularly large and there's no > > value in breaking things into multiple partitions for no reason. > > > > Maybe to have the ability to restore or reinstall the system without > bothering /home? > >
Re: Memory allocation failed during fsck of large EXT4 filesystem
On Mon, Jul 05, 2021 at 02:21:05PM -0700, Thomas D. Dean wrote: On 7/5/21 1:54 PM, Michael Stone wrote: On Mon, Jul 05, 2021 at 12:53:39PM +0300, IL Ka wrote: 7TB seems like too much for one partition imho. Consider splitting it into the parts That's silly. It's 2021; 7TB isn't particularly large and there's no value in breaking things into multiple partitions for no reason. Maybe to have the ability to restore or reinstall the system without bothering /home? I've never found that to be of much practical value. YMMV. Anyway, if you have reasons to partition, go for it--but partitioning because a filesystem is arbitrarily "too big" isn't a reason. (Especially since, in context, it's a logical volume so presumably the OP has already decided how to allocate space and if they wanted a separate home they could have just made a lv for it.)
Re: Memory allocation failed during fsck of large EXT4 filesystem
On 7/5/21 1:54 PM, Michael Stone wrote: On Mon, Jul 05, 2021 at 12:53:39PM +0300, IL Ka wrote: 7TB seems like too much for one partition imho. Consider splitting it into the parts That's silly. It's 2021; 7TB isn't particularly large and there's no value in breaking things into multiple partitions for no reason. Maybe to have the ability to restore or reinstall the system without bothering /home?
Re: Memory allocation failed during fsck of large EXT4 filesystem
On Mon, Jul 05, 2021 at 12:53:39PM +0300, IL Ka wrote: 7TB seems like too much for one partition imho. Consider splitting it into the parts That's silly. It's 2021; 7TB isn't particularly large and there's no value in breaking things into multiple partitions for no reason.
Re: Memory allocation failed during fsck of large EXT4 filesystem
Reiner Buehl [2021-07-05 10:21:13] wrote: > Hi all, > I have a corrupt EXT4 filesystem where fsck.ext4 fails with the error > message: > > Error storing directory block information (inode=366740508, block=0, > num=406081): Memory allocation failed [...] > The system has 4GB of memory and a 8GB swap partition. The filesystem has > 7TB. Is there a quick way to enlarge the swap space to help fsck.ext4 to > finish the repair? I do not have any unused partitions but have space for > swap on other filesystems if that is possible. I think you should report this as a bug in e2fsck. While 7TB is significantly larger than the partitions I have, 8GB of swap should still be plenty for that (my first 1TB disk was connected to a machine with 64MB of RAM (an asus wl-700ge) and fsck was slow but it worked), so I suspect the error is not in the lack of memory space. Stefan
Re: Memory allocation failed during fsck of large EXT4 filesystem
On 7/5/2021 4:30 AM, Reiner Buehl wrote: Hi all, I have a corrupt EXT4 filesystem where fsck.ext4 fails with the error message: Error storing directory block information (inode=366740508, block=0, num=406081): Memory allocation failed /dev/vg_data/lv_mpg: * FILE SYSTEM WAS MODIFIED * e2fsck: aborted /dev/vg_data/lv_mpg: * FILE SYSTEM WAS MODIFIED * The system has 4GB of memory and a 8GB swap partition. The filesystem has 7TB. Is there a quick way to enlarge the swap space to help fsck.ext4 to finish the repair? I do not have any unused partitions but have space for swap on other filesystems if that is possible. Are you sure it's not a ulimit issue? Does the ulimit command return unlimited?
Re: Memory allocation failed during fsck of large EXT4 filesystem
On Mon, Jul 5, 2021 at 5:17 PM Reiner Buehl wrote: > It seems swap is not the solution: Even after adding a 50G swap file, I > still get the same error message and the swap usage stats from collectd > show that max swap usage was not more than just 2G. > > btw, do you use 32 or 64bit os?
Re: Memory allocation failed during fsck of large EXT4 filesystem
It seems swap is not the solution: Even after adding a 50G swap file, I still get the same error message and the swap usage stats from collectd show that max swap usage was not more than just 2G. I will now try if the scratch_files stanza makes a difference. Am Mo., 5. Juli 2021 um 11:50 Uhr schrieb Thomas Schmitt : > Hi, > > Reiner Buehl wrote: > > Is there a quick way to enlarge the swap space > > According to old memories of mine you may create a large, non-sparse file > as you would do for a virtual disk. E.g. by mkfile (which seems not to be > in Debian) or qemu-img (from qemu-utils): > > qemu-img create "$swap_file_path" 8G > > Set ownership and access rights of "$swap_file_path" so that no > unprivileged users can spy. > > Then you tell the system to use it for swapping > > swapon "$swap_file_path" > > > > fsck.ext4 fails with the error message: > > Error storing directory block information (inode=366740508, block=0, > > num=406081): Memory allocation failed > > According to > > https://codesearch.debian.net/search?q=package%3Ae2fsprogs+Memory+allocation+failed > this message is emitted if error code EXT2_ET_NO_MEMORY was returned. > This error code indeed occurs if memory allocating system calls fail. > In these cases i would expect that more virtual memory could help. > > > > > But i see questionable occurences of EXT2_ET_NO_MEMORY which get triggered > by bad data. In these cases no extra memory can help: > > Halfways correct is its use to mark an insane request for a memory array > which would exceed the maximum number that can be stored in unsigned long > integer variables. > > There are possible misattributions of that error code if get_icount_el() > returns 0 to set_inode_count() for reasons of bad data. > > https://sources.debian.org/src/e2fsprogs/1.46.2-2/lib/ext2fs/icount.c/?hl=461#L461 > > https://sources.debian.org/src/e2fsprogs/1.46.2-2/lib/ext2fs/icount.c/?hl=388#L388 > (in line 496 the same return value leads to ENOENT.) > > In > > https://sources.debian.org/src/e2fsprogs/1.46.2-2/lib/ext2fs/hashmap.c/?hl=33#L33 > i see a potential memory fault by using the calloc(3) return without > checking it for NULL. (A caller of ext2fs_hashmap_create() would later > throw EXT2_ET_NO_MEMORY if the program did not crash yet.) > > Return value 0 from get_refcount_el() is converted to EXT2_ET_NO_MEMORY > in ea_refcount_increment(), although get_refcount_el() did not attempt to > allocate memory. > > It stays a riddle from where e2fsprogs links sparse_file_new(). I find it > only as Android C++ call. Whatever, if it fails then EXT2_ET_NO_MEMORY > can be returned by its caller io_manager_configure(), which seems not > restricted to Android. > > > Have a nice day :) > > Thomas > >
Re: Memory allocation failed during fsck of large EXT4 filesystem
Hi, Reiner Buehl wrote: > Is there a quick way to enlarge the swap space According to old memories of mine you may create a large, non-sparse file as you would do for a virtual disk. E.g. by mkfile (which seems not to be in Debian) or qemu-img (from qemu-utils): qemu-img create "$swap_file_path" 8G Set ownership and access rights of "$swap_file_path" so that no unprivileged users can spy. Then you tell the system to use it for swapping swapon "$swap_file_path" > fsck.ext4 fails with the error message: > Error storing directory block information (inode=366740508, block=0, > num=406081): Memory allocation failed According to https://codesearch.debian.net/search?q=package%3Ae2fsprogs+Memory+allocation+failed this message is emitted if error code EXT2_ET_NO_MEMORY was returned. This error code indeed occurs if memory allocating system calls fail. In these cases i would expect that more virtual memory could help. But i see questionable occurences of EXT2_ET_NO_MEMORY which get triggered by bad data. In these cases no extra memory can help: Halfways correct is its use to mark an insane request for a memory array which would exceed the maximum number that can be stored in unsigned long integer variables. There are possible misattributions of that error code if get_icount_el() returns 0 to set_inode_count() for reasons of bad data. https://sources.debian.org/src/e2fsprogs/1.46.2-2/lib/ext2fs/icount.c/?hl=461#L461 https://sources.debian.org/src/e2fsprogs/1.46.2-2/lib/ext2fs/icount.c/?hl=388#L388 (in line 496 the same return value leads to ENOENT.) In https://sources.debian.org/src/e2fsprogs/1.46.2-2/lib/ext2fs/hashmap.c/?hl=33#L33 i see a potential memory fault by using the calloc(3) return without checking it for NULL. (A caller of ext2fs_hashmap_create() would later throw EXT2_ET_NO_MEMORY if the program did not crash yet.) Return value 0 from get_refcount_el() is converted to EXT2_ET_NO_MEMORY in ea_refcount_increment(), although get_refcount_el() did not attempt to allocate memory. It stays a riddle from where e2fsprogs links sparse_file_new(). I find it only as Android C++ call. Whatever, if it fails then EXT2_ET_NO_MEMORY can be returned by its caller io_manager_configure(), which seems not restricted to Android. Have a nice day :) Thomas
Re: Memory allocation failed during fsck of large EXT4 filesystem
> Error storing directory block information (inode=366740508, block=0, > num=406081): Memory allocation failed > Try ``scratch_files`` https://manpages.debian.org/buster/e2fsprogs/e2fsck.conf.5.en.html This stanza controls when e2fsck will attempt to use scratch files to reduce the need for memory. > The system has 4GB of memory and a 8GB swap partition. The filesystem has > 7TB. Is there a quick way to enlarge the swap space to help fsck.ext4 to > finish the repair? > I do not have any unused partitions but have space for swap on other > filesystems if that is possible. > You can create swap on any free partition, see https://manpages.debian.org/buster/util-linux/mkswap.8.en.html https://manpages.debian.org/buster/mount/swapon.8.en.html 7TB seems like too much for one partition imho. Consider splitting it into the parts > >
Memory allocation failed during fsck of large EXT4 filesystem
Hi all, I have a corrupt EXT4 filesystem where fsck.ext4 fails with the error message: Error storing directory block information (inode=366740508, block=0, num=406081): Memory allocation failed /dev/vg_data/lv_mpg: * FILE SYSTEM WAS MODIFIED * e2fsck: aborted /dev/vg_data/lv_mpg: * FILE SYSTEM WAS MODIFIED * The system has 4GB of memory and a 8GB swap partition. The filesystem has 7TB. Is there a quick way to enlarge the swap space to help fsck.ext4 to finish the repair? I do not have any unused partitions but have space for swap on other filesystems if that is possible.
