[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
Ubuntu 21.10 (Impish Indri) has reached end of life, so this bug will not be fixed for that specific release. ** Changed in: linux (Ubuntu Impish) Status: Triaged => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Released Status in linux source package in Focal: Fix Released Status in linux source package in Groovy: Won't Fix Status in linux source package in Hirsute: Fix Released Status in linux source package in Impish: Won't Fix Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
This bug was fixed in the package linux - 4.15.0-154.161 --- linux (4.15.0-154.161) bionic; urgency=medium * bionic/linux: 4.15.0-154.161 -proposed tracker (LP: #1938411) * Potential reverts of 4.19.y stable changes in 18.04 (LP: #1938537) - SAUCE: Revert "locking/mutex: clear MUTEX_FLAGS if wait_list is empty due to signal" - SAUCE: Revert "drm/amd/amdgpu: fix refcount leak" * Packaging resync (LP: #1786013) - [Packaging] resync getabis - [Packaging] update helper scripts - update dkms package versions * btrfs: Automatic balance returns -EUCLEAN and leads to forced readonly filesystem (LP: #1934709) // CVE-2019-19036 - btrfs: Validate child tree block's level and first key - btrfs: Detect unbalanced tree with empty leaf before crashing btree operations * btrfs: Automatic balance returns -EUCLEAN and leads to forced readonly filesystem (LP: #1934709) - Revert "btrfs: Detect unbalanced tree with empty leaf before crashing btree operations" - Revert "btrfs: Validate child tree block's level and first key" - btrfs: Only check first key for committed tree blocks - btrfs: Fix wrong first_key parameter in replace_path * Enable fib-onlink-tests.sh and msg_zerocopy.sh in kselftests/net on Bionic (LP: #1934759) - selftests: Add fib-onlink-tests.sh to TEST_PROGS - selftests: net: use TEST_PROGS_EXTENDED - selftests/net: enable msg_zerocopy test - SAUCE: selftests: Make fib-onlink-tests.sh executable * Kernel oops due to uninitialized list on kernfs (kernfs_kill_sb) (LP: #1934175) - kernfs: deal with kernfs_fill_super() failures - unfuck sysfs_mount() * large_dir in ext4 broken (LP: #1933074) - SAUCE: ext4: fix directory index node split corruption * btrfs: Attempting to balance a nearly full filesystem with relocated root nodes fails (LP: #1933172) // CVE-2019-19036 - btrfs: reloc: fix reloc root leak and NULL pointer dereference * btrfs: Attempting to balance a nearly full filesystem with relocated root nodes fails (LP: #1933172) - Revert "btrfs: reloc: fix reloc root leak and NULL pointer dereference" * Pixel format change broken for Elgato Cam Link 4K (LP: #1932367) - (upstream) media: uvcvideo: Fix pixel format change for Elgato Cam Link 4K * Bionic update: upstream stable patchset 2021-06-23 (LP: #1933375) - net: usb: cdc_ncm: don't spew notifications - efi: Allow EFI_MEMORY_XP and EFI_MEMORY_RO both to be cleared - efi: cper: fix snprintf() use in cper_dimm_err_location() - vfio/pci: Fix error return code in vfio_ecap_init() - vfio/pci: zap_vma_ptes() needs MMU - vfio/platform: fix module_put call in error flow - ipvs: ignore IP_VS_SVC_F_HASHED flag when adding service - HID: pidff: fix error return code in hid_pidff_init() - HID: i2c-hid: fix format string mismatch - netfilter: nfnetlink_cthelper: hit EBUSY on updates if size mismatches - ieee802154: fix error return code in ieee802154_add_iface() - ieee802154: fix error return code in ieee802154_llsec_getparams() - Bluetooth: fix the erroneous flush_work() order - Bluetooth: use correct lock to prevent UAF of hdev object - net: caif: added cfserl_release function - net: caif: add proper error handling - net: caif: fix memory leak in caif_device_notify - net: caif: fix memory leak in cfusbl_device_notify - ALSA: timer: Fix master timer notification - ext4: fix bug on in ext4_es_cache_extent as ext4_split_extent_at failed - pid: take a reference when initializing `cad_pid` - ocfs2: fix data corruption by fallocate - nfc: fix NULL ptr dereference in llcp_sock_getname() after failed connect - btrfs: fix error handling in btrfs_del_csums - btrfs: fixup error handling in fixup_inode_link_counts - mm, hugetlb: fix simple resv_huge_pages underflow on UFFDIO_COPY - selftests/bpf: make 'dubious pointer arithmetic' test useful - bnxt_en: Remove the setting of dev_port. - KVM: SVM: Truncate GPR value for DR and CR accesses in !64-bit mode - sched/fair: Optimize select_idle_cpu - xen-pciback: redo VF placement in the virtual topology - ALSA: usb: update old-style static const declaration - nl80211: validate key indexes for cfg80211_registered_device - x86/apic: Mark _all_ legacy interrupts when IO/APIC is missing - btrfs: return errors from btrfs_del_csums in cleanup_ref_head - KVM: arm64: Fix debug register indexing -- Kleber Sacilotto de Souza Fri, 30 Jul 2021 14:39:24 +0200 ** Changed in: linux (Ubuntu Bionic) Status: Fix Committed => Fix Released ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2019-19036 -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
This bug was fixed in the package linux - 5.11.0-31.33 --- linux (5.11.0-31.33) hirsute; urgency=medium * hirsute/linux: 5.11.0-31.33 -proposed tracker (LP: #1939553) * REGRESSION: shiftfs lets sendfile fail with EINVAL (LP: #1939301) - SAUCE: shiftfs: fix sendfile() invocations linux (5.11.0-26.28) hirsute; urgency=medium * Packaging resync (LP: #1786013) - update dkms package versions * large_dir in ext4 broken (LP: #1933074) - SAUCE: ext4: fix directory index node split corruption * Add l2tp.sh in net from ubuntu_kernel_selftests back (LP: #1934293) - Revert "UBUNTU: SAUCE: selftests/net -- disable l2tp.sh test" * icmp_redirect.sh in net from ubuntu_kernel_selftests failed on F-OEM-5.6 / F-OEM-5.10 / F-OEM-5.13 / F / G / H (LP: #1880645) - selftests: icmp_redirect: support expected failures * Mute/mic LEDs no function on some HP platfroms (LP: #1934878) - ALSA: hda/realtek: fix mute/micmute LEDs for HP ProBook 450 G8 - ALSA: hda/realtek: fix mute/micmute LEDs for HP ProBook 445 G8 - ALSA: hda/realtek: fix mute/micmute LEDs for HP ProBook 630 G8 * [SRU][OEM-5.10/H] Fix HDMI output issue on Intel TGL GPU (LP: #1934864) - drm/i915: Fix HAS_LSPCON macro for platforms between GEN9 and GEN10 * mute/micmute LEDs no function on HP EliteBook 830 G8 Notebook PC (LP: #1934239) - ALSA: hda/realtek: fix mute/micmute LEDs for HP EliteBook 830 G8 Notebook PC * ubuntu-host driver lacks lseek ops (LP: #1934110) - ubuntu-host: add generic lseek op * ubuntu_kernel_selftests ftrace fails on arm64 F / aws-5.8 / amd64 F azure-5.8 (LP: #1927749) - selftests/ftrace: fix event-no-pid on 1-core machine * Hirsute update: upstream stable patchset 2021-06-29 (LP: #1934012) - proc: Track /proc/$pid/attr/ opener mm_struct - ASoC: max98088: fix ni clock divider calculation - ASoC: amd: fix for pcm_read() error - spi: Fix spi device unregister flow - spi: spi-zynq-qspi: Fix stack violation bug - bpf: Forbid trampoline attach for functions with variable arguments - net/nfc/rawsock.c: fix a permission check bug - usb: cdns3: Fix runtime PM imbalance on error - ASoC: Intel: bytcr_rt5640: Add quirk for the Glavey TM800A550L tablet - ASoC: Intel: bytcr_rt5640: Add quirk for the Lenovo Miix 3-830 tablet - vfio-ccw: Reset FSM state to IDLE inside FSM - vfio-ccw: Serialize FSM IDLE state with I/O completion - ASoC: sti-sas: add missing MODULE_DEVICE_TABLE - spi: sprd: Add missing MODULE_DEVICE_TABLE - usb: chipidea: udc: assign interrupt number to USB gadget structure - isdn: mISDN: netjet: Fix crash in nj_probe: - bonding: init notify_work earlier to avoid uninitialized use - netlink: disable IRQs for netlink_lock_table() - net: mdiobus: get rid of a BUG_ON() - cgroup: disable controllers at parse time - wq: handle VM suspension in stall detection - net/qla3xxx: fix schedule while atomic in ql_sem_spinlock - RDS tcp loopback connection can hang - net:sfc: fix non-freed irq in legacy irq mode - scsi: bnx2fc: Return failure if io_req is already in ABTS processing - scsi: vmw_pvscsi: Set correct residual data length - scsi: hisi_sas: Drop free_irq() of devm_request_irq() allocated irq - scsi: target: qla2xxx: Wait for stop_phase1 at WWN removal - net: macb: ensure the device is available before accessing GEMGXL control registers - net: appletalk: cops: Fix data race in cops_probe1 - net: dsa: microchip: enable phy errata workaround on 9567 - nvme-fabrics: decode host pathing error for connect - MIPS: Fix kernel hang under FUNCTION_GRAPH_TRACER and PREEMPT_TRACER - dm verity: fix require_signatures module_param permissions - bnx2x: Fix missing error code in bnx2x_iov_init_one() - nvme-tcp: remove incorrect Kconfig dep in BLK_DEV_NVME - nvmet: fix false keep-alive timeout when a controller is torn down - powerpc/fsl: set fsl,i2c-erratum-a004447 flag for P2041 i2c controllers - powerpc/fsl: set fsl,i2c-erratum-a004447 flag for P1010 i2c controllers - spi: Don't have controller clean up spi device before driver unbind - spi: Cleanup on failure of initial setup - i2c: mpc: Make use of i2c_recover_bus() - i2c: mpc: implement erratum A-004447 workaround - ALSA: seq: Fix race of snd_seq_timer_open() - ALSA: firewire-lib: fix the context to call snd_pcm_stop_xrun() - spi: bcm2835: Fix out-of-bounds access with more than 4 slaves - Revert "ACPI: sleep: Put the FACS table after using it" - drm: Fix use-after-free read in drm_getunique() - drm: Lock pointer access in drm_master_release() - perf/x86/intel/uncore: Fix M2M event umask for Ice Lake server - KVM: X86: MMU: Use the correct inherited permissions to get shadow page - kvm: avoid speculation-based attacks from out-of-range memslot accesses - staging: rtl8723bs: Fix
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
This bug was fixed in the package linux - 5.4.0-81.91 --- linux (5.4.0-81.91) focal; urgency=medium * Packaging resync (LP: #1786013) - update dkms package versions * large_dir in ext4 broken (LP: #1933074) - SAUCE: ext4: fix directory index node split corruption * Some test in kselftest/net on focal source tree were not tested at all (LP: #1934282) - selftests/net: add missing tests to Makefile * curtin: install flash-kernel in arm64 UEFI unexpected (LP: #1918427) - [Packaging] Allow grub-efi-arm* to satisfy recommends on ARM * Add l2tp.sh in net from ubuntu_kernel_selftests back (LP: #1934293) - Revert "UBUNTU: SAUCE: selftests/net -- disable l2tp.sh test" * icmp_redirect.sh in net from ubuntu_kernel_selftests failed on F-OEM-5.6 / F-OEM-5.10 / F-OEM-5.13 / F / G / H (LP: #1880645) - selftests: icmp_redirect: support expected failures * Focal update: v5.4.128 upstream stable release (LP: #1934179) - dmaengine: ALTERA_MSGDMA depends on HAS_IOMEM - dmaengine: QCOM_HIDMA_MGMT depends on HAS_IOMEM - dmaengine: stedma40: add missing iounmap() on error in d40_probe() - afs: Fix an IS_ERR() vs NULL check - mm/memory-failure: make sure wait for page writeback in memory_failure - kvm: LAPIC: Restore guard to prevent illegal APIC register access - batman-adv: Avoid WARN_ON timing related checks - net: ipv4: fix memory leak in netlbl_cipsov4_add_std - vrf: fix maximum MTU - net: rds: fix memory leak in rds_recvmsg - net: lantiq: disable interrupt before sheduling NAPI - udp: fix race between close() and udp_abort() - rtnetlink: Fix regression in bridge VLAN configuration - net/sched: act_ct: handle DNAT tuple collision - net/mlx5e: Remove dependency in IPsec initialization flows - net/mlx5e: Fix page reclaim for dead peer hairpin - net/mlx5: Consider RoCE cap before init RDMA resources - net/mlx5e: allow TSO on VXLAN over VLAN topologies - net/mlx5e: Block offload of outer header csum for UDP tunnels - netfilter: synproxy: Fix out of bounds when parsing TCP options - sch_cake: Fix out of bounds when parsing TCP options and header - alx: Fix an error handling path in 'alx_probe()' - net: stmmac: dwmac1000: Fix extended MAC address registers definition - net: make get_net_ns return error if NET_NS is disabled - qlcnic: Fix an error handling path in 'qlcnic_probe()' - netxen_nic: Fix an error handling path in 'netxen_nic_probe()' - net: qrtr: fix OOB Read in qrtr_endpoint_post - ptp: improve max_adj check against unreasonable values - net: cdc_ncm: switch to eth%d interface naming - lantiq: net: fix duplicated skb in rx descriptor ring - net: usb: fix possible use-after-free in smsc75xx_bind - net: fec_ptp: fix issue caused by refactor the fec_devtype - net: ipv4: fix memory leak in ip_mc_add1_src - net/af_unix: fix a data-race in unix_dgram_sendmsg / unix_release_sock - be2net: Fix an error handling path in 'be_probe()' - net: hamradio: fix memory leak in mkiss_close - net: cdc_eem: fix tx fixup skb leak - cxgb4: fix wrong shift. - bnxt_en: Rediscover PHY capabilities after firmware reset - bnxt_en: Call bnxt_ethtool_free() in bnxt_init_one() error path - icmp: don't send out ICMP messages with a source address of 0.0.0.0 - net: ethernet: fix potential use-after-free in ec_bhf_remove - regulator: bd70528: Fix off-by-one for buck123 .n_voltages setting - ASoC: rt5659: Fix the lost powers for the HDA header - spi: stm32-qspi: Always wait BUSY bit to be cleared in stm32_qspi_wait_cmd() - pinctrl: ralink: rt2880: avoid to error in calls is pin is already enabled - radeon: use memcpy_to/fromio for UVD fw upload - hwmon: (scpi-hwmon) shows the negative temperature properly - can: bcm: fix infoleak in struct bcm_msg_head - can: bcm/raw/isotp: use per module netdevice notifier - can: j1939: fix Use-after-Free, hold skb ref while in use - can: mcba_usb: fix memory leak in mcba_usb - usb: core: hub: Disable autosuspend for Cypress CY7C65632 - tracing: Do not stop recording cmdlines when tracing is off - tracing: Do not stop recording comms if the trace file is being read - tracing: Do no increment trace_clock_global() by one - PCI: Mark TI C667X to avoid bus reset - PCI: Mark some NVIDIA GPUs to avoid bus reset - PCI: aardvark: Don't rely on jiffies while holding spinlock - PCI: aardvark: Fix kernel panic during PIO transfer - PCI: Add ACS quirk for Broadcom BCM57414 NIC - PCI: Work around Huawei Intelligent NIC VF FLR erratum - KVM: x86: Immediately reset the MMU context when the SMM flag is cleared - ARCv2: save ABI registers across signal handling - x86/process: Check PF_KTHREAD and not current->mm for kernel threads - x86/pkru: Write hardware init value to PKRU when xstate is init - x86/fpu:
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
The Groovy Gorilla has reached end of life, so this bug will not be fixed for that release ** Changed in: linux (Ubuntu Groovy) Status: Fix Committed => Won't Fix -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Won't Fix Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
Tested hirsute 5.11.0-26-generic, test passes fine. PASSED. ** Tags removed: verification-needed-bionic verification-needed-focal verification-needed-hirsute ** Tags added: verification-done-bionic verification-done-focal verification-done-hirsute -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
Tested focal 5.4.0-81-generic, test passes fine. PASSED. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
Tested bionic 4.15.0-152-generic, test passes fine. PASSED. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-bionic -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- bionic' to 'verification-done-bionic'. If the problem still exists, change the tag 'verification-needed-bionic' to 'verification-failed- bionic'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- hirsute' to 'verification-done-hirsute'. If the problem still exists, change the tag 'verification-needed-hirsute' to 'verification-failed- hirsute'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-hirsute ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
** Changed in: linux (Ubuntu Bionic) Status: Triaged => Fix Committed ** Changed in: linux (Ubuntu Focal) Status: Triaged => Fix Committed ** Changed in: linux (Ubuntu Groovy) Status: Triaged => Fix Committed ** Changed in: linux (Ubuntu Hirsute) Status: Triaged => Fix Committed -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Fix Committed Status in linux source package in Focal: Fix Committed Status in linux source package in Groovy: Fix Committed Status in linux source package in Hirsute: Fix Committed Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
** Changed in: linux (Ubuntu Bionic) Importance: Undecided => High ** Changed in: linux (Ubuntu Focal) Importance: Undecided => High ** Changed in: linux (Ubuntu Focal) Status: New => Triaged ** Changed in: linux (Ubuntu Groovy) Importance: Undecided => High ** Changed in: linux (Ubuntu Groovy) Status: New => Triaged ** Changed in: linux (Ubuntu Hirsute) Importance: Undecided => High ** Changed in: linux (Ubuntu Hirsute) Status: New => Triaged ** Changed in: linux (Ubuntu Impish) Importance: Undecided => High ** Changed in: linux (Ubuntu Impish) Status: New => Triaged -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: Triaged Status in linux source package in Bionic: Triaged Status in linux source package in Focal: Triaged Status in linux source package in Groovy: Triaged Status in linux source package in Hirsute: Triaged Status in linux source package in Impish: Triaged Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help :
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
** Changed in: linux (Ubuntu Bionic) Assignee: Stefan Bader (smb) => Colin Ian King (colin-king) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: New Status in linux source package in Bionic: Triaged Status in linux source package in Focal: New Status in linux source package in Groovy: New Status in linux source package in Hirsute: New Status in linux source package in Impish: New Bug description: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of thousands of empty files will trip the issue. It seems to occur more frequently with memory pressure and smaller block sizes, e.g.: sudo mkdir -p /mnt/tmpfs /mnt/storage sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F sudo mount /mnt/tmpfs/ext4.img /mnt/storage and compile and run the attached C program (see https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c) that quickly populates /mnt/storage with empty files. Without the fix this will terminate with an -EEXIST or -EUCLEAN error on the file creation after several tens of thousands of files. [Where problems could occur] This changes the behaviour of the directory indexing hashing so there is a regression potential that this may introduce subsequent index hashing issues when needed (or not) to do a split. This patch seems to cover all the necessary cases, so I believe this risk is relatively low. I have also tested this on all the kernel series in the SRU with 21,000,000 files so I am confident we have enough test coverage to show the fix is OK. -- I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
C source to reproduce the problem as fast as possible. The -r option will remove the files. ** Description changed: + == SRU, Bionic, Focal, Groovy, Hirsute, Impish == + + [Impact] + + Creating millions of files on ext4 partition with large_dir support by + touching them will eventually trip an ext4 leaf node issue in the index + hash. This occurs more frequently when also using smaller block sizes + and ends up either with a EXIST or EUCLEAN failure. + + This occurs on the restart condition when performing do_split. + + [ Fix ] + + The fix protects do_split() from the restart condition, making it safe + from both current and future ordering of goto statements in earlier + sections of the code. + + The fix is from a patch sent upstream and cc'd to Ted Tso but didn't + appear on the ext4 mailing list presumably because it got marked as + SPAM. + + [ Test Case ] + + Without the fix touching tens of thousands of empty files will trip the + issue. It seems to occur more frequently with memory pressure and + smaller block sizes, e.g.: + + sudo mkdir -p /mnt/tmpfs /mnt/storage + sudo mount -t tmpfs -o size=9000M tmpfs /mnt/tmpfs + sudo dd if=/dev/urandom of=/mnt/tmpfs/ext4.img bs=1M + sudo mkfs.ext4 -O large_dir -N 2100 -O dir_index /mnt/tmpfs/ext4.img -b 1024 -F + sudo mount /mnt/tmpfs/ext4.img /mnt/storage + + and compile and run the attached C program that quickly populates + /mnt/storage with empty files. Without the fix this will terminate with + an -EEXIST or -EUCLEAN error on the file creation after several tens of + thousands of files. + + [Where problems could occur] + + This changes the behaviour of the directory indexing hashing so there is + a regression potential that this may introduce subsequent index hashing + issues when needed (or not) to do a split. This patch seems to cover + all the necessary cases, so I believe this risk is relatively low. I + have also tested this on all the kernel series in the SRU with + 21,000,000 files so I am confident we have enough test coverage to show + the fix is OK. + + -- + I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do - i=$(( $i + 1 )); - (( $i % 1000 == 0 )) && echo $i; - touch file_$i.dat || break; + i=$(( $i + 1 )); + (( $i % 1000 == 0 )) && echo $i; + touch file_$i.dat || break; done - Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block - Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. ** Also affects: linux (Ubuntu Impish) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Focal) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Groovy) Importance: Undecided Status: New ** Also affects: linux (Ubuntu Hirsute) Importance: Undecided Status: New ** Attachment added: "C source to touch millions of files" https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+attachment/5509402/+files/touch.c ** Description changed: == SRU, Bionic, Focal, Groovy, Hirsute, Impish == [Impact] Creating millions of files on ext4 partition with large_dir support by touching them will eventually trip an ext4 leaf node issue in the index hash. This occurs more frequently when also using smaller block sizes and ends up either with a EXIST or EUCLEAN failure. This occurs on the restart condition when performing do_split. [ Fix ] The fix protects do_split() from the restart condition, making it safe from both current and future ordering of goto statements in earlier sections of the code. The fix is from a patch sent upstream and cc'd to Ted Tso but didn't appear on the ext4 mailing list presumably because it got marked as SPAM. [ Test Case ] Without the fix touching tens of
[Kernel-packages] [Bug 1933074] Re: large_dir in ext4 broken
** Package changed: linux-signed (Ubuntu) => linux (Ubuntu) ** Also affects: linux (Ubuntu Bionic) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Bionic) Status: New => Triaged ** Changed in: linux (Ubuntu Bionic) Assignee: (unassigned) => Stefan Bader (smb) -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to linux in Ubuntu. https://bugs.launchpad.net/bugs/1933074 Title: large_dir in ext4 broken Status in linux package in Ubuntu: New Status in linux source package in Bionic: Triaged Bug description: I believe, I found a bug in ext4 in recent kernel versions. I stumbled across this while I was trying to restore a backup to a new VM. How to reproduce this bug: 1. Use a virtual/physical machine with "Ubuntu 18.04.5 LTS" and kernel version 4.15.0-144-generic. 2. add a secondary disk to hold the test files. 3. prepare and mount the filesystem with enabled 'large_dir' flag: mkfs.ext4 -m0 /dev/sdb1; tune2fs -O large_dir /dev/sdb1; mkdir /mnt/storage; mount /dev/sdb1 /mnt/storage; 4. change to directory and create approx. 16 mio files cd /mnt/storage; i=0; while (( $i < 2000 )); do i=$(( $i + 1 )); (( $i % 1000 == 0 )) && echo $i; touch file_$i.dat || break; done Expected behaviour: - 20 mio files shoud be created without error What happened instead: - The loop aborts with an error message: # 16263100 # touch: cannot touch 'file_16263173.dat': Structure needs cleaning - dmesg gives a little more details: # [Mon Jun 21 03:15:18 2021] EXT4-fs error (device sdb): dx_probe:855: inode #2: block 146221: comm touch: directory leaf block found instead of index block Additional notes: - This occurs on kernel version 4.15.0-144-generic - Not sure, but I believe one test was run on 4.15.0-143-generic and failed too. - Did not check against 4.15.0-142-generic - On 4.15.0-141-generic, the problem does not exist. Behaviour is as expected. To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1933074/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp
