[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
This bug was fixed in the package linux - 5.8.0-53.60 --- linux (5.8.0-53.60) groovy; urgency=medium * CVE-2021-3491 - io_uring: fix provide_buffers sign extension - io_uring: fix overflows checks in provide buffers - SAUCE: proc: Avoid mixing integer types in mem_rw() - SAUCE: io_uring: truncate lengths larger than MAX_RW_COUNT on provide buffers * CVE-2021-3490 - bpf: Fix a verifier failure with xor - SAUCE: bpf: verifier: fix ALU32 bounds tracking with bitwise ops * CVE-2021-3489 - SAUCE: bpf: ringbuf: deny reserve of buffers larger than ringbuf - SAUCE: bpf: prevent writable memory-mapping of read-only ringbuf pages -- Stefan Bader Thu, 06 May 2021 07:43:20 +0200 ** Changed in: linux (Ubuntu Groovy) Status: Fix Committed => Fix Released ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2021-3489 ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2021-3490 ** CVE added: https://cve.mitre.org/cgi-bin/cvename.cgi?name=2021-3491 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
This bug was fixed in the package linux - 5.4.0-73.82
---
linux (5.4.0-73.82) focal; urgency=medium
* focal/linux: 5.4.0-73.82 -proposed tracker (LP: #1923781)
* Packaging resync (LP: #1786013)
- update dkms package versions
* CIFS DFS entries not accessible with 5.4.0-71.74-generic (LP: #1923670)
- Revert "cifs: Set CIFS_MOUNT_USE_PREFIX_PATH flag on setting
cifs_sb->prepath."
* CVE-2021-29650
- Revert "netfilter: x_tables: Update remaining dereference to RCU"
- Revert "netfilter: x_tables: Switch synchronization to RCU"
- netfilter: x_tables: Use correct memory barriers.
* LRMv4: switch to signing nvidia modules via the Ubuntu Modules signing key
(LP: #1918134)
- [Packaging] dkms-build{,--nvidia-N} sync back from LRMv4
* 5.4 kernel: when iommu is on crashdump fails (LP: #1922738)
- iommu/vt-d: Refactor find_domain() helper
- iommu/vt-d: Add attach_deferred() helper
- iommu/vt-d: Move deferred device attachment into helper function
- iommu/vt-d: Do deferred attachment in iommu_need_mapping()
- iommu/vt-d: Remove deferred_attach_domain()
- iommu/vt-d: Simplify check in identity_mapping()
* Backport mlx5e fix for tunnel offload (LP: #1921769)
- net/mlx5e: Check tunnel offload is required before setting SWP
* Bcache bypasse writeback on caching device with fragmentation (LP: #1900438)
- bcache: consider the fragmentation when update the writeback rate
* Fix implicit declaration warnings for kselftests/memfd test on newer
releases (LP: #1910323)
- selftests/memfd: Fix implicit declaration warnings
* net/mlx5e: Add missing capability check for uplink follow (LP: #1921104)
- net/mlx5e: Add missing capability check for uplink follow
* [UBUNUT 21.04] s390/vtime: fix increased steal time accounting
(LP: #1921498)
- s390/vtime: fix increased steal time accounting
* Mute/Mic-mute LEDs are not work on HP 850/840/440 G8 Laptops (LP: #1920030)
- ALSA: hda/realtek: fix mute/micmute LEDs for HP 840 G8
- ALSA: hda/realtek: fix mute/micmute LEDs for HP 440 G8
- ALSA: hda/realtek: fix mute/micmute LEDs for HP 850 G8
* Focal update: v5.4.106 upstream stable release (LP: #1920246)
- uapi: nfnetlink_cthelper.h: fix userspace compilation error
- powerpc/pseries: Don't enforce MSI affinity with kdump
- ath9k: fix transmitting to stations in dynamic SMPS mode
- net: Fix gro aggregation for udp encaps with zero csum
- net: check if protocol extracted by virtio_net_hdr_set_proto is correct
- net: avoid infinite loop in mpls_gso_segment when mpls_hlen == 0
- sh_eth: fix TRSCER mask for SH771x
- can: skb: can_skb_set_owner(): fix ref counting if socket was closed
before
setting skb ownership
- can: flexcan: assert FRZ bit in flexcan_chip_freeze()
- can: flexcan: enable RX FIFO after FRZ/HALT valid
- can: flexcan: invoke flexcan_chip_freeze() to enter freeze mode
- can: tcan4x5x: tcan4x5x_init(): fix initialization - clear MRAM before
entering Normal Mode
- tcp: add sanity tests to TCP_QUEUE_SEQ
- netfilter: nf_nat: undo erroneous tcp edemux lookup
- netfilter: x_tables: gpf inside xt_find_revision()
- selftests/bpf: No need to drop the packet when there is no geneve opt
- selftests/bpf: Mask bpf_csum_diff() return value to 16 bits in
test_verifier
- samples, bpf: Add missing munmap in xdpsock
- ibmvnic: always store valid MAC address
- mt76: dma: do not report truncated frames to mac80211
- powerpc/603: Fix protection of user pages mapped with PROT_NONE
- mount: fix mounting of detached mounts onto targets that reside on shared
mounts
- cifs: return proper error code in statfs(2)
- Revert "mm, slub: consider rest of partial list if acquire_slab() fails"
- net: enetc: don't overwrite the RSS indirection table when initializing
- net/mlx4_en: update moderation when config reset
- net: stmmac: fix incorrect DMA channel intr enable setting of EQoS v4.10
- nexthop: Do not flush blackhole nexthops when loopback goes down
- net: sched: avoid duplicates in classes dump
- net: usb: qmi_wwan: allow qmimux add/del with master up
- netdevsim: init u64 stats for 32bit hardware
- cipso,calipso: resolve a number of problems with the DOI refcounts
- net: lapbether: Remove netif_start_queue / netif_stop_queue
- net: davicom: Fix regulator not turned off on failed probe
- net: davicom: Fix regulator not turned off on driver removal
- net: qrtr: fix error return code of qrtr_sendmsg()
- ixgbe: fail to create xfrm offload of IPsec tunnel mode SA
- net: stmmac: stop each tx channel independently
- net: stmmac: fix watchdog timeout during suspend/resume stress test
- selftests: forwarding: Fix race condition in mirror installation
- perf traceevent: Ensure read cmdlines are null terminated.
- net: hns3: fix query vlan mask
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
This bug was fixed in the package linux - 4.15.0-143.147
---
linux (4.15.0-143.147) bionic; urgency=medium
* bionic/linux: 4.15.0-143.147 -proposed tracker (LP: #1923811)
* CVE-2021-29650
- netfilter: x_tables: Use correct memory barriers.
* LRMv4: switch to signing nvidia modules via the Ubuntu Modules signing key
(LP: #1918134)
- [Packaging] dkms-build{,--nvidia-N} sync back from LRMv4
* Security-Fix Xen XSA 371 for Kernel 5.4.0-71 (LP: #1921902) //
CVE-2021-28688
- xen-blkback: don't leak persistent grants from xen_blkbk_map()
* CVE-2021-20292
- drm/ttm/nouveau: don't call tt destroy callback on alloc failure.
* CVE-2021-29264
- gianfar: fix jumbo packets+napi+rx overrun crash
* CVE-2021-29265
- usbip: fix stub_dev usbip_sockfd_store() races leading to gpf
* Bcache bypasse writeback on caching device with fragmentation (LP: #1900438)
- bcache: consider the fragmentation when update the writeback rate
* Bionic update: upstream stable patchset 2021-03-31 (LP: #1922124)
- net: usb: qmi_wwan: support ZTE P685M modem
- scripts: use pkg-config to locate libcrypto
- scripts: set proper OpenSSL include dir also for sign-file
- hugetlb: fix update_and_free_page contig page struct assumption
- drm/virtio: use kvmalloc for large allocations
- virtio/s390: implement virtio-ccw revision 2 correctly
- arm64 module: set plt* section addresses to 0x0
- arm64: Avoid redundant type conversions in xchg() and cmpxchg()
- arm64: cmpxchg: Use "K" instead of "L" for ll/sc immediate constraint
- arm64: Use correct ll/sc atomic constraints
- JFS: more checks for invalid superblock
- media: mceusb: sanity check for prescaler value
- xfs: Fix assert failure in xfs_setattr_size()
- smackfs: restrict bytes count in smackfs write functions
- net: fix up truesize of cloned skb in skb_prepare_for_shift()
- mm/hugetlb.c: fix unnecessary address expansion of pmd sharing
- net: bridge: use switchdev for port flags set through sysfs too
- dt-bindings: net: btusb: DT fix s/interrupt-name/interrupt-names/
- staging: fwserial: Fix error handling in fwserial_create
- x86/reboot: Add Zotac ZBOX CI327 nano PCI reboot quirk
- vt/consolemap: do font sum unsigned
- wlcore: Fix command execute failure 19 for wl12xx
- pktgen: fix misuse of BUG_ON() in pktgen_thread_worker()
- ath10k: fix wmi mgmt tx queue full due to race condition
- x86/build: Treat R_386_PLT32 relocation as R_386_PC32
- Bluetooth: Fix null pointer dereference in amp_read_loc_assoc_final_data
- staging: most: sound: add sanity check for function argument
- media: uvcvideo: Allow entities with no pads
- f2fs: handle unallocated section and zone on pinned/atgc
- parisc: Bump 64-bit IRQ stack size to 64 KB
- Xen/gnttab: handle p2m update errors on a per-slot basis
- xen-netback: respect gnttab_map_refs()'s return value
- zsmalloc: account the number of compacted pages correctly
- swap: fix swapfile read/write offset
- media: v4l: ioctl: Fix memory leak in video_usercopy
- PCI: Add a REBAR size quirk for Sapphire RX 5600 XT Pulse
- drm/amd/display: Guard against NULL pointer deref when get_i2c_info fails
- f2fs: fix to set/clear I_LINKABLE under i_lock
- btrfs: fix error handling in commit_fs_roots
- ALSA: hda/realtek: Add quirk for Clevo NH55RZQ
- ALSA: hda/realtek: Apply dual codec quirks for MSI Godlike X570 board
- btrfs: raid56: simplify tracking of Q stripe presence
- btrfs: fix raid6 qstripe kmap
- usbip: tools: fix build error for multiple definition
- ALSA: ctxfi: cthw20k2: fix mask on conf to allow 4 bits
- rsxx: Return -EFAULT if copy_to_user() fails
- dm table: fix iterate_devices based device capability checks
- dm table: fix DAX iterate_devices based device capability checks
- dm table: fix zoned iterate_devices based device capability checks
- iommu/amd: Fix sleeping in atomic in increase_address_space()
- mwifiex: pcie: skip cancel_work_sync() on reset failure path
- platform/x86: acer-wmi: Cleanup ACER_CAP_FOO defines
- platform/x86: acer-wmi: Cleanup accelerometer device handling
- platform/x86: acer-wmi: Add new force_caps module parameter
- platform/x86: acer-wmi: Add ACER_CAP_SET_FUNCTION_MODE capability flag
- platform/x86: acer-wmi: Add support for SW_TABLET_MODE on Switch devices
- platform/x86: acer-wmi: Add ACER_CAP_KBD_DOCK quirk for the Aspire Switch
10E SW3-016
- PCI: Add function 1 DMA alias quirk for Marvell 9215 SATA controller
- misc: eeprom_93xx46: Add quirk to support Microchip 93LC46B eeprom
- drm/msm/a5xx: Remove overwriting A5XX_PC_DBG_ECO_CNTL register
- Revert "zram: close udev startup race condition as default groups"
- HID: mf: add support for 0079:1846 Mayflash/Dragonrise USB Gamecube
Adapter
* Bionic update:
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
I used the same steps to verify it and can confirm it's succeeded in bionic-proposed too fio --name=test1 --filename /dev/bcache0 --direct 1 --rw=randrw --bs=32k,4k --ioengine=libaio --iodepth=1 Then I monitoring the cache_available_percent and writeback rate. I no longer see the cache_available_percent dropped to 30 and writeback rate stucked at 4k. I can see the rate got accelerated while the dirty buckets hit the writeback_rate_*_fp_term, which is expected -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Tags added: verification-done-bionic -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
I used the same steps to verify it and can conconfirm it's succeeded fio --name=test1 --filename /dev/bcache0 --direct 1 --rw=randrw --bs=32k,4k --ioengine=libaio --iodepth=1 Then I monitoring the cache_available_percent and writeback rate. I no longer see the cache_available_percent dropped to 30 and writeback rate stucked at 4k. I can see the rate got accelerated while the dirty buckets hit the writeback_rate_*_fp_term, which is expected ** Tags removed: verification-needed-groovy ** Tags added: verification-done-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- groovy' to 'verification-done-groovy'. If the problem still exists, change the tag 'verification-needed-groovy' to 'verification-failed- groovy'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-groovy -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
I've done the verification of the focal-proposed kernel. I used below fio command to cause the bcache fragmentation: fio --name=test1 --filename /dev/bcache0 --direct 1 --rw=randrw --bs=32k,4k --ioengine=libaio --iodepth=1 Then I monitoring the cache_available_percent and writeback rate. I no longer see the cache_available_percent dropped to 30 and writeback rate stucked at 4k. I can see the rate got accelerated while the dirty buckets hit the writeback_rate_*_fp_term, which is expected ** Tags removed: verification-needed-focal ** Tags added: verification-done-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
This bug is awaiting verification that the kernel in -proposed solves the problem. Please test the kernel and update this bug with the results. If the problem is solved, change the tag 'verification-needed- focal' to 'verification-done-focal'. If the problem still exists, change the tag 'verification-needed-focal' to 'verification-failed-focal'. If verification is not done by 5 working days from today, this fix will be dropped from the source code, and this bug will be closed. See https://wiki.ubuntu.com/Testing/EnableProposed for documentation how to enable and use -proposed. Thank you! ** Tags added: verification-needed-focal -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
This bug was fixed in the package linux - 5.11.0-14.15 --- linux (5.11.0-14.15) hirsute; urgency=medium * hirsute/linux: 5.11.0-14.15 -proposed tracker (LP: #1923103) * Packaging resync (LP: #1786013) - update dkms package versions * Include Infiniband Peer Memory interface (LP: #1923104) - SAUCE: RDMA/core: Introduce peer memory interface * Hirsute update: v5.11.12 upstream stable release (LP: #1923069) - arm64: mm: correct the inside linear map range during hotplug check - virtiofs: Fail dax mount if device does not support it - ext4: shrink race window in ext4_should_retry_alloc() - ext4: fix bh ref count on error paths - fs: nfsd: fix kconfig dependency warning for NFSD_V4 - rpc: fix NULL dereference on kmalloc failure - iomap: Fix negative assignment to unsigned sis->pages in iomap_swapfile_activate - ASoC: rt1015: fix i2c communication error - ASoC: rt5640: Fix dac- and adc- vol-tlv values being off by a factor of 10 - ASoC: rt5651: Fix dac- and adc- vol-tlv values being off by a factor of 10 - ASoC: sgtl5000: set DAP_AVC_CTRL register to correct default value on probe - ASoC: es8316: Simplify adc_pga_gain_tlv table - ASoC: soc-core: Prevent warning if no DMI table is present - ASoC: cs42l42: Fix Bitclock polarity inversion - ASoC: cs42l42: Fix channel width support - ASoC: cs42l42: Fix mixer volume control - ASoC: cs42l42: Always wait at least 3ms after reset - NFSD: fix error handling in NFSv4.0 callbacks - ASoC: mediatek: mt8192: fix tdm out data is valid on rising edge - kernel: freezer should treat PF_IO_WORKER like PF_KTHREAD for freezing - vhost: Fix vhost_vq_reset() - io_uring: fix ->flags races by linked timeouts - io_uring: halt SQO submission on ctx exit - scsi: st: Fix a use after free in st_open() - scsi: qla2xxx: Fix broken #endif placement - staging: comedi: cb_pcidas: fix request_irq() warn - staging: comedi: cb_pcidas64: fix request_irq() warn - ASoC: rt5659: Update MCLK rate in set_sysclk() - ASoC: rt711: add snd_soc_component remove callback - thermal/core: Add NULL pointer check before using cooling device stats - locking/ww_mutex: Simplify use_ww_ctx & ww_ctx handling - locking/ww_mutex: Fix acquire/release imbalance in ww_acquire_init()/ww_acquire_fini() - nvmet-tcp: fix kmap leak when data digest in use - io_uring: imply MSG_NOSIGNAL for send[msg]()/recv[msg]() calls - Revert "PM: ACPI: reboot: Use S5 for reboot" - nouveau: Skip unvailable ttm page entries - static_call: Align static_call_is_init() patching condition - ext4: do not iput inode under running transaction in ext4_rename() - io_uring: call req_set_fail_links() on short send[msg]()/recv[msg]() with MSG_WAITALL - net: mvpp2: fix interrupt mask/unmask skip condition - mptcp: deliver ssk errors to msk - mptcp: fix poll after shutdown - mptcp: init mptcp request socket earlier - mptcp: add a missing retransmission timer scheduling - flow_dissector: fix TTL and TOS dissection on IPv4 fragments - mptcp: fix DATA_FIN processing for orphaned sockets - mptcp: provide subflow aware release function - can: dev: move driver related infrastructure into separate subdir - net: introduce CAN specific pointer in the struct net_device - mptcp: fix race in release_cb - net: bonding: fix error return code of bond_neigh_init() - mptcp: fix bit MPTCP_PUSH_PENDING tests - can: tcan4x5x: fix max register value - brcmfmac: clear EAP/association status bits on linkdown events - ath11k: add ieee80211_unregister_hw to avoid kernel crash caused by NULL pointer - netdevsim: dev: Initialize FIB module after debugfs - iwlwifi: pcie: don't disable interrupts for reg_lock - ath10k: hold RCU lock when calling ieee80211_find_sta_by_ifaddr() - net: ethernet: aquantia: Handle error cleanup of start on open - appletalk: Fix skb allocation size in loopback case - net: ipa: remove two unused register definitions - net: ipa: use a separate pointer for adjusted GSI memory - net: ipa: fix register write command validation - net: wan/lmc: unregister device when no matching device is found - net: 9p: advance iov on empty read - bpf: Remove MTU check in __bpf_skb_max_len - ACPI: tables: x86: Reserve memory occupied by ACPI tables - ACPI: processor: Fix CPU0 wakeup in acpi_idle_play_dead() - ACPI: scan: Fix _STA getting called on devices with unmet dependencies - ALSA: usb-audio: Apply sample rate quirk to Logitech Connect - ALSA: hda: Re-add dropped snd_poewr_change_state() calls - ALSA: hda: Add missing sanity checks in PM prepare/complete callbacks - ALSA: hda/realtek: call alc_update_headset_mode() in hp_automute_hook - xtensa: fix uaccess-related livelock in do_page_fault - xtensa: move coprocessor_flush to
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Changed in: linux (Ubuntu Bionic) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Focal) Status: In Progress => Fix Committed ** Changed in: linux (Ubuntu Groovy) Status: In Progress => Fix Committed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Changed in: linux (Ubuntu Bionic) Importance: Medium => High ** Changed in: linux (Ubuntu Focal) Importance: Medium => High ** Changed in: linux (Ubuntu Groovy) Importance: Medium => High -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Also affects: linux (Ubuntu Hirsute) Importance: Undecided Assignee: dongdong tao (taodd) Status: Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Also affects: linux (Ubuntu Groovy) Importance: Undecided Status: New ** Changed in: linux (Ubuntu Groovy) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Groovy) Status: New => In Progress ** Changed in: linux (Ubuntu Groovy) Assignee: (unassigned) => dongdong tao (taodd) ** Changed in: linux (Ubuntu Focal) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Focal) Status: New => In Progress ** Changed in: linux (Ubuntu Bionic) Importance: Undecided => Medium ** Changed in: linux (Ubuntu Bionic) Status: New => In Progress -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Description changed:
SRU Justification:
[Impact]
- This bug in bcache [insert correct area] affects I/O performance on all
versions of the kernel [correct versions affected]. It is particularly negative
on ceph if used with bcache.
+ This bug in bcache affects I/O performance on all versions of the kernel
[correct versions affected]. It is particularly negative on ceph if used with
bcache.
Write I/O latency would suddenly go to around 1 second from around 10 ms
when hitting this issue and would easily be stuck there for hours or
even days, especially bad for ceph on bcache architecture. This would
make ceph extremely slow and make the entire cloud almost unusable.
The root cause is that the dirty bucket had reached the 70 percent
threshold, thus causing all writes to go direct to the backing HDD
device. It might be fine if it actually had a lot of dirty data, but
this happens when dirty data has not even reached over 10 percent, due
to having high memory fragmentation. What makes it worse is that the
writeback rate might be still at minimum value (8) due to the writeback
percent not reached, so it takes ages for bcache to really reclaim
enough dirty buckets to get itself out of this situation.
[Fix]
* 71dda2a5625f31bc3410cb69c3d31376a2b66f28 “bcache: consider the
fragmentation when update the writeback rate”
- The current way to calculate the writeback rate only considered the dirty
sectors.
+ The current way to calculate the writeback rate only considered the dirty
sectors.
This usually works fine when memory fragmentation is not high, but it will
give us an unreasonably low writeback rate when we are in the situation that a
few dirty sectors have consumed a lot of dirty buckets. In some cases, the
dirty buckets reached CUTOFF_WRITEBACK_SYNC (i.e., stopped writeback) while
the dirty data (sectors) had not even reached the writeback_percent threshold
(i.e., started writeback). In that situation, the writeback rate will still be
the minimum value (8*512 = 4KB/s), thus it will cause all the writes to bestuck
in a non-writeback mode because of the slow writeback.
We accelerate the rate in 3 stages with different aggressiveness:
- the first stage starts when dirty buckets percent reach above
BCH_WRITEBACK_FRAGMENT_THRESHOLD_LOW (50),
+ the first stage starts when dirty buckets percent reach above
BCH_WRITEBACK_FRAGMENT_THRESHOLD_LOW (50),
the second is BCH_WRITEBACK_FRAGMENT_THRESHOLD_MID (57),
- the third is BCH_WRITEBACK_FRAGMENT_THRESHOLD_HIGH (64).
+ the third is BCH_WRITEBACK_FRAGMENT_THRESHOLD_HIGH (64).
By default the first stage tries to writeback the amount of dirty data
in one bucket (on average) in (1 / (dirty_buckets_percent - 50)) seconds,
the second stage tries to writeback the amount of dirty data in one bucket
in (1 / (dirty_buckets_percent - 57)) * 100 milliseconds, the third
stage tries to writeback the amount of dirty data in one bucket in
(1 / (dirty_buckets_percent - 64)) milliseconds.
The initial rate at each stage can be controlled by 3 configurable
- parameters:
+ parameters:
writeback_rate_fp_term_{low|mid|high}
They are by default 1, 10, 1000, chosen based on testing and production
data, detailed below.
A. When it comes to the low stage, it is still far from the 70%
-threshold, so we only want to give it a little bit push by setting the
-term to 1, it means the initial rate will be 170 if the fragment is 6,
-it is calculated by bucket_size/fragment, this rate is very small,
-but still much more reasonable than the minimum 8.
-For a production bcache with non-heavy workload, if the cache device
-is bigger than 1 TB, it may take hours to consume 1% buckets,
-so it is very possible to reclaim enough dirty buckets in this stage,
-thus to avoid entering the next stage.
+ threshold, so we only want to give it a little bit push by setting the
+ term to 1, it means the initial rate will be 170 if the fragment is 6,
+ it is calculated by bucket_size/fragment, this rate is very small,
+ but still much more reasonable than the minimum 8.
+ For a production bcache with non-heavy workload, if the cache device
+ is bigger than 1 TB, it may take hours to consume 1% buckets,
+ so it is very possible to reclaim enough dirty buckets in this stage,
+ thus to avoid entering the next stage.
B. If the dirty buckets ratio didn’t turn around during the first stage,
-it comes to the mid stage, then it is necessary for mid stage
-to be more aggressive than low stage, so the initial rate is chosen
-to be 10 times more than the low stage, which means 1700 as the initial
-rate if the fragment is 6. This is a normal rate
-we usually see for a normal workload when writeback happens
-because of writeback_percent.
+ it comes to the mid stage, then it is necessary for mid stage
+ to be more aggressive than
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Description changed:
- Hello,
+ SRU Justification:
- An upstream bug has been opened on the matter for quite some time now [0].
- I can reproduce easily on our production compute node instance, which are
trusty host with xenial hwe kernels (4.15.0-101-generic).
- However due to heavy backport and such, doing real tracing is a bit hard
there.
+ [Impact]
+ This bug in bcache [insert correct area] affects I/O performance on all
versions of the kernel [correct versions affected]. It is particularly negative
on ceph if used with bcache.
- I was able to reproduce the behavior on a hwe-bionic kernel as well.
- Since most of our critical deployments use bcache, I think this is a kinda
nasty bug to have.
+ Write I/O latency would suddenly go to around 1 second from around 10 ms
+ when hitting this issue and would easily be stuck there for hours or
+ even days, especially bad for ceph on bcache architecture. This would
+ make ceph extremely slow and make the entire cloud almost unusable.
- Reproducing the issue is relatively easy with the script provided in the bug
[1].
- The script used to capture the stats is this one [2].
+ The root cause is that the dirty bucket had reached the 70 percent
+ threshold, thus causing all writes to go direct to the backing HDD
+ device. It might be fine if it actually had a lot of dirty data, but
+ this happens when dirty data has not even reached over 10 percent, due
+ to having high memory fragmentation. What makes it worse is that the
+ writeback rate might be still at minimum value (8) due to the writeback
+ percent not reached, so it takes ages for bcache to really reclaim
+ enough dirty buckets to get itself out of this situation.
- [0]: https://bugzilla.kernel.org/show_bug.cgi?id=206767
- [1]: https://pastebin.ubuntu.com/p/YnnvvSRhXK/
- [2]: https://pastebin.ubuntu.com/p/XfVpzg32sN/
+ [Fix]
+
+ * 71dda2a5625f31bc3410cb69c3d31376a2b66f28 “bcache: consider the
+ fragmentation when update the writeback rate”
+
+ The current way to calculate the writeback rate only considered the dirty
sectors.
+ This usually works fine when memory fragmentation is not high, but it will
give us an unreasonably low writeback rate when we are in the situation that a
few dirty sectors have consumed a lot of dirty buckets. In some cases, the
dirty buckets reached CUTOFF_WRITEBACK_SYNC (i.e., stopped writeback) while
the dirty data (sectors) had not even reached the writeback_percent threshold
(i.e., started writeback). In that situation, the writeback rate will still be
the minimum value (8*512 = 4KB/s), thus it will cause all the writes to bestuck
in a non-writeback mode because of the slow writeback.
+
+ We accelerate the rate in 3 stages with different aggressiveness:
+ the first stage starts when dirty buckets percent reach above
BCH_WRITEBACK_FRAGMENT_THRESHOLD_LOW (50),
+ the second is BCH_WRITEBACK_FRAGMENT_THRESHOLD_MID (57),
+ the third is BCH_WRITEBACK_FRAGMENT_THRESHOLD_HIGH (64).
+
+ By default the first stage tries to writeback the amount of dirty data
+ in one bucket (on average) in (1 / (dirty_buckets_percent - 50)) seconds,
+ the second stage tries to writeback the amount of dirty data in one bucket
+ in (1 / (dirty_buckets_percent - 57)) * 100 milliseconds, the third
+ stage tries to writeback the amount of dirty data in one bucket in
+ (1 / (dirty_buckets_percent - 64)) milliseconds.
+
+ The initial rate at each stage can be controlled by 3 configurable
+ parameters:
+
+ writeback_rate_fp_term_{low|mid|high}
+
+ They are by default 1, 10, 1000, chosen based on testing and production
+ data, detailed below.
+
+ A. When it comes to the low stage, it is still far from the 70%
+threshold, so we only want to give it a little bit push by setting the
+term to 1, it means the initial rate will be 170 if the fragment is 6,
+it is calculated by bucket_size/fragment, this rate is very small,
+but still much more reasonable than the minimum 8.
+For a production bcache with non-heavy workload, if the cache device
+is bigger than 1 TB, it may take hours to consume 1% buckets,
+so it is very possible to reclaim enough dirty buckets in this stage,
+thus to avoid entering the next stage.
+
+ B. If the dirty buckets ratio didn’t turn around during the first stage,
+it comes to the mid stage, then it is necessary for mid stage
+to be more aggressive than low stage, so the initial rate is chosen
+to be 10 times more than the low stage, which means 1700 as the initial
+rate if the fragment is 6. This is a normal rate
+we usually see for a normal workload when writeback happens
+because of writeback_percent.
+
+ C. If the dirty buckets ratio didn't turn around during the low and mid
+stages, it comes to the third stage, and it is the last chance that
+we can turn around to avoid the horrible cutoff writeback sync issue,
+then we choose 100 times more aggressive than the mid stage, that
+
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
Here is the latest updated patch and the progress being made in upstream https://marc.info/?l=linux-bcache=160981605206306=1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
I've just submitted a patch[1] to upstream for review to help with this problem. The key is to speed up the writeback rate when the fragmentation is high. Here are the comments from the patch: Current way to calculate the writeback rate only considered the dirty sectors, this usually works fine when the fragmentation is not high, but it will give us unreasonable small rate when we are under a situation that very few dirty sectors consumed a lot dirty buckets. In some case, the dirty bucekts can reached to CUTOFF_WRITEBACK_SYNC while the dirty data(sectors) noteven reached the writeback_percent, the writeback rate will still be the minimum value (4k), thus it will cause all the writes to be stucked in a non-writeback mode because of the slow writeback. This patch will try to accelerate the writeback rate when the fragmentation is high. It calculate the propotional_scaled value based on below: (dirty_sectors / writeback_rate_p_term_inverse) * fragment As we can see, the higher fragmentation will result a larger proportional_scaled value, thus cause a larger writeback rate. The fragment value is calculated based on below: (dirty_buckets * bucket_size) / dirty_sectors If you think about it, the value of fragment will be always inside [1, bucket_size]. This patch only considers the fragmentation when the number of dirty_buckets reached to a dirty threshold(configurable by writeback_fragment_percent, default is 50), so bcache will remain the original behaviour before the dirty buckets reached the threshold. [1] https://marc.info/?l=linux-bcache=160441418209114=1 -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
I tried to run the apport-collect on one of the server we can see the
issue on :
Waiting to hear from Launchpad about your decision...
*** Collecting problem information
The collected information can be sent to the developers to improve the
application. This might take a few minutes.
..dpkg-query: no packages found matching linux
..
Traceback (most recent call last):
File "/usr/bin/apport-cli", line 370, in
if not app.run_argv():
File "/usr/lib/python2.7/dist-packages/apport/ui.py", line 666, in run_argv
return self.run_update_report()
File "/usr/lib/python2.7/dist-packages/apport/ui.py", line 564, in
run_update_report
self.report.add_proc_environ()
File "/usr/lib/python2.7/dist-packages/apport/report.py", line 577, in
add_proc_environ
proc_pid_fd = os.open('/proc/%s' % pid, os.O_RDONLY | os.O_PATH |
os.O_DIRECTORY)
AttributeError: 'module' object has no attribute 'O_PATH'
Error in sys.excepthook:
Traceback (most recent call last):
File "/usr/lib/python2.7/dist-packages/apport_python_hook.py", line 109, in
apport_excepthook
pr.add_proc_info(extraenv=['PYTHONPATH', 'PYTHONHOME'])
File "/usr/lib/python2.7/dist-packages/apport/report.py", line 507, in
add_proc_info
proc_pid_fd = os.open('/proc/%s' % pid, os.O_RDONLY | os.O_PATH |
os.O_DIRECTORY)
AttributeError: 'module' object has no attribute 'O_PATH'
But since the bug has already been marked as "Confirmed".
--
You received this bug notification because you are a member of Ubuntu
Bugs, which is subscribed to Ubuntu.
https://bugs.launchpad.net/bugs/1900438
Title:
Bcache bypasse writeback on caching device with fragmentation
To manage notifications about this bug go to:
https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions
--
ubuntu-bugs mailing list
ubuntu-bugs@lists.ubuntu.com
https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1900438] Re: Bcache bypasse writeback on caching device with fragmentation
** Changed in: linux (Ubuntu) Status: Incomplete => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1900438 Title: Bcache bypasse writeback on caching device with fragmentation To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/linux/+bug/1900438/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
