[Bug 1883676] Re: ZFS performance drops suddenly
After a lot of tests on AWS with generic and AWS kernels and on non-AWS bare-metal systems with the AWS and generic kernels I discovered that one gets I/O throttling on AWS instances regardless of using ZFS or not. I ran a really simple test writing and reading to the raw device and found the *same* I/O throttling characteristics. This is due to a feature known as "EBS I/O throttling", see: https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ebs-io-characteristics.html and https://aws.amazon.com/ebs/features/ This is not a zfs issue per se, or a kernel issue, but due to I/O throttling with the cloud provider. Marking this as won't fix. ** Changed in: zfs-linux (Ubuntu) Status: Incomplete => Won't Fix -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
@Andrey, it is noted that ZFS I/O can slow down when ZFS volumes fill up, noticeably at around 80-85% full capacity writes can slow dramatically while free space is being found. Is this possibly the reason for the speed bottleneck? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
Hi Andrey, does setting zfs_arc_max resolve your issue? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
I've been examining the zfs_arc_max setting to see if this can stabilize the I/O performance. You may like to try setting this as a proportion of your available memory to see if it improves things. For example, with a machine with 8GB of memory, I set this to 2GB and my test system went from being able to reproduce your issue to one where the I/O performance remained constant for hundreds of fio tests. For example, to set 2GB arc max use: echo 2147483648 | sudo tee /sys/module/zfs/parameters/zfs_arc_max sync echo 3 | sudo tee /proc/sys/vm/drop_caches If it is set too low it impacts on cache hits and this reduces I/O rates. If it is set too high you hit the issue you are seeing because of cache contention between ZFS and the block layer. I suspect the only way to find the sweet spot for your use case is with some careful experimentation. Don't rely on synthetic tests to set this, real world I/O patterns are best to use for this kind of tuning. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1883676] Re: ZFS performance drops suddenly
1. free totalusedfree shared buff/cache available Mem:8025436 1255616 6457736 11780 312084 6512296 Swap: 1425404 0 1425404 2. cat /proc/spl/kstat/zfs/arcstats 12 1 0x01 98 26656 4085867100 20862566338548 nametype data hits48977275 misses 420770 demand_data_hits47154775 demand_data_misses 413760 demand_metadata_hits41810210 demand_metadata_misses 44292 prefetch_data_hits 4112 prefetch_data_misses4719 prefetch_metadata_hits 412178 prefetch_metadata_misses41999 mru_hits44041554 mru_ghost_hits 40 mfu_hits44923504 mfu_ghost_hits 40 deleted 431 mutex_miss 40 access_skip 49 evict_skip 4124 evict_not_enough40 evict_l2_cached 40 evict_l2_eligible 4396800 evict_l2_ineligible 48192 evict_l2_skip 40 hash_elements 420337 hash_elements_max 422033 hash_collisions 482157 hash_chains 4196 hash_chain_max 42 p 42054511616 c 44109023232 c_min 4256813952 c_max 44109023232 size41856513648 compressed_size 41349638144 uncompressed_size 41738882048 overhead_size 4446370304 hdr_size47828496 data_size 41709473792 metadata_size 486534656 dbuf_size 49437376 dnode_size 423086368 bonus_size 420152960 anon_size 4438411264 anon_evictable_data 40 anon_evictable_metadata 40 mru_size41186710528 mru_evictable_data 4908280320 mru_evictable_metadata 44658688 mru_ghost_size 40 mru_ghost_evictable_data40 mru_ghost_evictable_metadata40 mfu_size4170886656 mfu_evictable_data 4134954496 mfu_evictable_metadata 43134976 mfu_ghost_size 40 mfu_ghost_evictable_data40 mfu_ghost_evictable_metadata40 l2_hits 40 l2_misses 40 l2_feeds40 l2_rw_clash 40 l2_read_bytes 40 l2_write_bytes 40 l2_writes_sent 40 l2_writes_done 40 l2_writes_error 40 l2_writes_lock_retry40 l2_evict_lock_retry 40 l2_evict_reading40 l2_evict_l1cached 40 l2_free_on_write40 l2_abort_lowmem 40 l2_cksum_bad40 l2_io_error 40 l2_size 40 l2_asize40 l2_hdr_size 40 memory_throttle_count 40 memory_direct_count 40 memory_indirect_count 40 memory_all_bytes48218046464 memory_free_bytes 45230358528 memory_available_bytes 35101953024 arc_no_grow 40 arc_tempreserve 40 arc_loaned_bytes40 arc_prune 40 arc_meta_used 4147039856 arc_meta_limit 43081767424 arc_dnode_limit 4308176742 arc_meta_max4152895280 arc_meta_min416777216 async_upgrade_sync 459 demand_hit_predictive_prefetch 4484 demand_hit_prescient_prefetch 40 arc_need_free 40 arc_sys_free4128406976 arc_raw_size40 This behavior was observed on more than one machine, both small and large (the large was on AWS). Thanks, Andrey Gelman On Mon, Jul 13, 2020 at 5:45 PM Colin Ian King <1883...@bugs.launchpad.net> wrote: > It would be interesting in knowing the following data: > > 1. How much memory does the machine have when booted? > > run: free > > 2. What the arcstats look like when it goes slowly > > use: cat /proc/spl/kstat/zfs/arcstats > > thanks > > ** Changed in: zfs-linux (Ubuntu) >Status: Confirmed => Incomplete >
[Bug 1883676] Re: ZFS performance drops suddenly
It would be interesting in knowing the following data: 1. How much memory does the machine have when booted? run: free 2. What the arcstats look like when it goes slowly use: cat /proc/spl/kstat/zfs/arcstats thanks ** Changed in: zfs-linux (Ubuntu) Status: Confirmed => Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
Can you inform me the exact test configuration you were using? -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
Oh, no problem, I can see that in the bug report. Apologies. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
** Changed in: zfs-linux (Ubuntu) Importance: Undecided => High ** Changed in: zfs-linux (Ubuntu) Assignee: (unassigned) => Colin Ian King (colin-king) -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1883676] Re: ZFS performance drops suddenly
Reproducible. ** Changed in: zfs-linux (Ubuntu) Status: New => Confirmed -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1883676 Title: ZFS performance drops suddenly To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1883676/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
