Hi Eric, I applied these patches, built them in a ppa and then did a clean install of zfsutils-linux from -proposed and then upgraded to the packages in the ppa. However, your patches break the upgrade:
king@ubuntu:~$ dpkg -l | grep zfsutils-linux ii zfsutils-linux 0.6.5.6-0ubuntu11 amd64 Native OpenZFS management utilities for Linux king@ubuntu:~$ sudo add-apt-repository ppa:colin-king/zfs-0.6.5.7-lp1574342 [sudo] password for king: More info: https://launchpad.net/~colin-king/+archive/ubuntu/zfs-0.6.5.7-lp1574342 Press [ENTER] to continue or ctrl-c to cancel adding it gpg: keyring `/tmp/tmp3bjyv0sl/secring.gpg' created gpg: keyring `/tmp/tmp3bjyv0sl/pubring.gpg' created gpg: requesting key B0F01786 from hkp server keyserver.ubuntu.com gpg: /tmp/tmp3bjyv0sl/trustdb.gpg: trustdb created gpg: key XXXXXXXXX: public key "Launchpad PPA for Colin King" imported gpg: Total number processed: 1 gpg: imported: 1 (RSA: 1) OK king@ubuntu:~$ sudo apt-get update Hit:1 http://security.ubuntu.com/ubuntu xenial-security InRelease Hit:2 http://gb.archive.ubuntu.com/ubuntu xenial InRelease Get:3 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial InRelease [17.6 kB] Hit:4 http://gb.archive.ubuntu.com/ubuntu xenial-updates InRelease Hit:5 http://gb.archive.ubuntu.com/ubuntu xenial-proposed InRelease Hit:6 http://gb.archive.ubuntu.com/ubuntu xenial-backports InRelease Get:7 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 Packages [3,024 B] Get:8 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main i386 Packages [3,032 B] Get:9 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main Translation-en [1,584 B] Fetched 25.2 kB in 1s (21.5 kB/s) Reading package lists... Done king@ubuntu:~$ sudo apt-get dist-upgrade Reading package lists... Done Building dependency tree Reading state information... Done Calculating upgrade... Done The following package was automatically installed and is no longer required: linux-image-extra-4.3.0-7-generic Use 'sudo apt autoremove' to remove it. The following NEW packages will be installed libpython-stdlib python python-minimal python2.7 python2.7-minimal The following packages will be upgraded: libnvpair1linux libuutil1linux libzfs2linux libzpool2linux zfs-doc zfs-zed zfsutils-linux 7 to upgrade, 5 to newly install, 0 to remove and 0 not to upgrade. Need to get 2,610 kB of archives. After this operation, 4,986 kB of additional disk space will be used. Do you want to continue? [Y/n] Get:1 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python2.7-minimal amd64 2.7.12-1~16.04 [1,294 kB] Get:2 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libuutil1linux amd64 0.6.5.6-0ubuntu12 [27.5 kB] Get:3 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libzpool2linux amd64 0.6.5.6-0ubuntu12 [384 kB] Get:4 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libzfs2linux amd64 0.6.5.6-0ubuntu12 [106 kB] Get:5 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 zfs-zed amd64 0.6.5.6-0ubuntu12 [29.8 kB] Get:6 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 zfsutils-linux amd64 0.6.5.6-0ubuntu12 [290 kB] Get:7 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 libnvpair1linux amd64 0.6.5.6-0ubuntu12 [23.4 kB] Get:8 http://ppa.launchpad.net/colin-king/zfs-0.6.5.7-lp1574342/ubuntu xenial/main amd64 zfs-doc all 0.6.5.6-0ubuntu12 [58.2 kB] Get:9 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python-minimal amd64 2.7.11-1 [28.2 kB] Get:10 http://gb.archive.ubuntu.com/ubuntu xenial-updates/main amd64 python2.7 amd64 2.7.12-1~16.04 [224 kB] Get:11 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 libpython-stdlib amd64 2.7.11-1 [7,656 B] Get:12 http://gb.archive.ubuntu.com/ubuntu xenial/main amd64 python amd64 2.7.11-1 [137 kB] Fetched 2,610 kB in 1s (2,399 kB/s) Selecting previously unselected package python2.7-minimal. (Reading database ... 128781 files and directories currently installed.) Preparing to unpack .../python2.7-minimal_2.7.12-1~16.04_amd64.deb ... Unpacking python2.7-minimal (2.7.12-1~16.04) ... Selecting previously unselected package python-minimal. Preparing to unpack .../python-minimal_2.7.11-1_amd64.deb ... Unpacking python-minimal (2.7.11-1) ... Selecting previously unselected package python2.7. Preparing to unpack .../python2.7_2.7.12-1~16.04_amd64.deb ... Unpacking python2.7 (2.7.12-1~16.04) ... Selecting previously unselected package libpython-stdlib:amd64. Preparing to unpack .../libpython-stdlib_2.7.11-1_amd64.deb ... Unpacking libpython-stdlib:amd64 (2.7.11-1) ... Processing triggers for man-db (2.7.5-1) ... Processing triggers for mime-support (3.59ubuntu1) ... Setting up python2.7-minimal (2.7.12-1~16.04) ... Linking and byte-compiling packages for runtime python2.7... Setting up python-minimal (2.7.11-1) ... Selecting previously unselected package python. (Reading database ... 128837 files and directories currently installed.) Preparing to unpack .../python_2.7.11-1_amd64.deb ... Unpacking python (2.7.11-1) ... Preparing to unpack .../libuutil1linux_0.6.5.6-0ubuntu12_amd64.deb ... Unpacking libuutil1linux (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... Preparing to unpack .../libzpool2linux_0.6.5.6-0ubuntu12_amd64.deb ... Unpacking libzpool2linux (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... Preparing to unpack .../libzfs2linux_0.6.5.6-0ubuntu12_amd64.deb ... Unpacking libzfs2linux (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... Preparing to unpack .../zfs-zed_0.6.5.6-0ubuntu12_amd64.deb ... Unpacking zfs-zed (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... Preparing to unpack .../zfsutils-linux_0.6.5.6-0ubuntu12_amd64.deb ... Unpacking zfsutils-linux (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... dpkg: error processing archive /var/cache/apt/archives/zfsutils-linux_0.6.5.6-0ubuntu12_amd64.deb (--unpack): trying to overwrite '/usr/sbin/zed', which is also in package zfs-zed 0.6.5.6-0ubuntu12 dpkg-deb: error: subprocess paste was killed by signal (Broken pipe) zfs-import-cache.service is a disabled or a static unit, not starting it. zfs-import-scan.service is a disabled or a static unit, not starting it. zfs-mount.service is a disabled or a static unit, not starting it. Preparing to unpack .../libnvpair1linux_0.6.5.6-0ubuntu12_amd64.deb ... Unpacking libnvpair1linux (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... Preparing to unpack .../zfs-doc_0.6.5.6-0ubuntu12_all.deb ... Unpacking zfs-doc (0.6.5.6-0ubuntu12) over (0.6.5.6-0ubuntu11) ... Processing triggers for man-db (2.7.5-1) ... Processing triggers for libc-bin (2.23-0ubuntu3) ... Processing triggers for initramfs-tools (0.122ubuntu8.1) ... update-initramfs: Generating /boot/initrd.img-4.4.0-34-generic Processing triggers for systemd (229-4ubuntu7) ... Processing triggers for ureadahead (0.100.0-19) ... Errors were encountered while processing: /var/cache/apt/archives/zfsutils-linux_0.6.5.6-0ubuntu12_amd64.deb E: Sub-process /usr/bin/dpkg returned an error code (1) king@ubuntu:~$ I think the problem is the change to the path: override_dh_auto_configure: @# Build the userland, but don't build the kernel modules. dh_auto_configure -- \ - --bindir=/bin \ + --bindir=/usr/sbin \ --sbindir=/sbin \ --libdir=/lib \ ..which is causing the zed issues. It's heading to 9pm UK time, so I'm running out of time to rebuild and test this hypothesis. -- You received this bug notification because you are a member of Kernel Packages, which is subscribed to zfs-linux in Ubuntu. https://bugs.launchpad.net/bugs/1574342 Title: Ship arcstat.py and arc_summary.py with zfsutils-linux Status in zfs-linux package in Ubuntu: In Progress Status in zfs-linux source package in Xenial: In Progress Status in zfs-linux source package in Yakkety: In Progress Bug description: [Impact] zfsutils-linux missing some important tools. Python3 compatible scripts : arc_summary.py, arcstat.py and dbufstat.py found in src_code/cmd/ are not taken into account during compilation. * arc_summary.py : Provides a summary of the statistics https://github.com/zfsonlinux/zfs/blob/master/cmd/arc_summary * arcstat.py : Print out ZFS ARC Statistics exported via kstat(1) https://github.com/zfsonlinux/zfs/tree/master/cmd/arcstat * dbufstat.py : Print out statistics for all cached dmu buffers https://github.com/zfsonlinux/zfs/blob/master/cmd/dbufstat [Test Case] # rmadison zfsutils-linux ... zfsutils-linux | 0.6.5.6-0ubuntu10 | xenial-updates ... # List contents of a deb package. $ dpkg -c zfsutils-linux_0.6.5.6-0ubuntu10_amd64.deb | egrep "arcstat|arc_summary|dbufstat" # You will notice there are not present. [Regression Potential] None expected, debian package[1] has already made the change to build the python scripts. [1] Package: zfsutils-linux Source: zfs-linux Version: 0.6.5.7-1 Architecture: amd64 Maintainer: Debian ZFS on Linux maintainers <pkg-zfsonlinux-de...@lists.alioth.debian.org> # List contents of a deb package. $ dpkg -c zfsutils-linux_0.6.5.7-1_amd64.deb | egrep -i "arc|dbufstat" -rwxr-xr-x root/root 40751 2016-05-31 03:40 ./bin/arc_summary.py -rwxr-xr-x root/root 13057 2016-05-31 03:40 ./bin/arcstat.py -rwxr-xr-x root/root 18718 2016-05-31 03:40 ./bin/dbufstat.py [Other Info] * This is based on the following Debian commits : Debian git commits === commit 730989397063955d60a1845003cfb7433eba1e52 Author: Aron Xu <a...@debian.org> Date: Tue Apr 26 17:01:29 2016 +0800 Add dh-python to b-d -- commit 814413118fb88de3ce596a87a8a011f50c225fdd Author: Aron Xu <a...@debian.org> Date: Sun Nov 22 15:44:16 2015 +0800 Add python related dependencies === * Despite what Debian did by adding theses script under /bin. I'll add them under /usr/sbin, which is more appropriate. * Unfortunately manpage for theses 3 program seems limited. There is none in Debian nor git upstream repo. The only manpage I found is for arcstat online : http://linux.die.net/man/1/arcstat [Original Description] arcstat.py and arc_summary.py are valuable tools to determine ZFS' ARC usage, it is not obvious why they are not included in zfsutils-linux. As ubuntu-minimal already depends on python3 it should be safe to assume python is available, or am I mistaken here? arcstat.py gives an iostat-like overview about ARC reads, hit rate, current and target size in regular intervals: # ./arcstat.py 1 time read miss miss% dmis dm% pmis pm% mmis mm% arcsz c 19:22:33 0 0 0 0 0 0 0 0 0 2.0G 7.8G 19:22:34 3 0 0 0 0 0 0 0 0 2.0G 7.8G 19:22:35 21 0 0 0 0 0 0 0 0 2.0G 7.8G ^C arc_summary.py shows a more detailed overview of the current ARC status and ZFS tunables: # ./arc_summary.py ------------------------------------------------------------------------ ZFS Subsystem Report Sun Apr 24 19:23:25 2016 ARC Summary: (HEALTHY) Memory Throttle Count: 0 ARC Misc: Deleted: 46 Mutex Misses: 0 Evict Skips: 0 ARC Size: 25.10% 1.95 GiB Target Size: (Adaptive) 100.00% 7.78 GiB Min Size (Hard Limit): 0.40% 32.00 MiB Max Size (High Water): 248:1 7.78 GiB ARC Size Breakdown: Recently Used Cache Size: 50.00% 3.89 GiB Frequently Used Cache Size: 50.00% 3.89 GiB ARC Hash Breakdown: Elements Max: 32.31k Elements Current: 99.78% 32.24k Collisions: 40.54k Chain Max: 3 Chains: 240 ARC Total accesses: 4.54m Cache Hit Ratio: 99.39% 4.51m Cache Miss Ratio: 0.61% 27.74k Actual Hit Ratio: 98.76% 4.48m Data Demand Efficiency: 99.73% 3.23m Data Prefetch Efficiency: 11.32% 6.41k CACHE HITS BY CACHE LIST: Anonymously Used: 0.64% 28.65k Most Recently Used: 21.90% 987.29k Most Frequently Used: 77.47% 3.49m Most Recently Used Ghost: 0.00% 0 Most Frequently Used Ghost: 0.00% 0 CACHE HITS BY DATA TYPE: Demand Data: 71.40% 3.22m Prefetch Data: 0.02% 725 Demand Metadata: 27.97% 1.26m Prefetch Metadata: 0.62% 27.92k CACHE MISSES BY DATA TYPE: Demand Data: 31.81% 8.82k Prefetch Data: 20.48% 5.68k Demand Metadata: 21.99% 6.10k Prefetch Metadata: 25.72% 7.13k File-Level Prefetch: (HEALTHY) DMU Efficiency: 36.39m Hit Ratio: 93.36% 33.97m Miss Ratio: 6.64% 2.42m Colinear: 2.42m Hit Ratio: 0.02% 505 Miss Ratio: 99.98% 2.42m Stride: 33.94m Hit Ratio: 100.00% 33.94m Miss Ratio: 0.00% 14 DMU Misc: Reclaim: 2.42m Successes: 2.46% 59.51k Failures: 97.54% 2.36m Streams: 35.39k +Resets: 0.05% 18 -Resets: 99.95% 35.37k Bogus: 0 ZFS Tunable: metaslab_debug_load 0 zfs_arc_min_prefetch_lifespan 0 zfetch_max_streams 8 zfs_nopwrite_enabled 1 zfetch_min_sec_reap 2 zfs_dbgmsg_enable 0 zfs_dirty_data_max_max_percent 25 zfs_arc_p_aggressive_disable 1 spa_load_verify_data 1 zfs_zevent_cols 80 zfs_dirty_data_max_percent 10 zfs_sync_pass_dont_compress 5 l2arc_write_max 8388608 zfs_vdev_scrub_max_active 2 zfs_vdev_sync_write_min_active 10 zvol_prefetch_bytes 131072 metaslab_aliquot 524288 zfs_no_scrub_prefetch 0 zfs_arc_shrink_shift 0 zfetch_block_cap 256 zfs_txg_history 0 zfs_delay_scale 500000 zfs_vdev_async_write_active_min_dirty_percent 30 metaslab_debug_unload 0 zfs_read_history 0 zvol_max_discard_blocks 16384 zfs_recover 0 l2arc_headroom 2 zfs_deadman_synctime_ms 1000000 zfs_scan_idle 50 zfs_free_min_time_ms 1000 zfs_dirty_data_max 1670300876 zfs_vdev_async_read_min_active 1 zfs_mg_noalloc_threshold 0 zfs_dedup_prefetch 0 zfs_vdev_max_active 1000 l2arc_write_boost 8388608 zfs_resilver_min_time_ms 3000 zfs_vdev_async_write_max_active 10 zil_slog_limit 1048576 zfs_prefetch_disable 0 zfs_resilver_delay 2 metaslab_lba_weighting_enabled 1 zfs_mg_fragmentation_threshold 85 l2arc_feed_again 1 zfs_zevent_console 0 zfs_immediate_write_sz 32768 zfs_dbgmsg_maxsize 4194304 zfs_free_leak_on_eio 0 zfs_deadman_enabled 1 metaslab_bias_enabled 1 zfs_arc_p_dampener_disable 1 zfs_object_mutex_size 64 zfs_metaslab_fragmentation_threshold 70 zfs_no_scrub_io 0 metaslabs_per_vdev 200 zfs_dbuf_state_index 0 zfs_vdev_sync_read_min_active 10 metaslab_fragmentation_factor_enabled 1 zvol_inhibit_dev 0 zfs_vdev_async_write_active_max_dirty_percent 60 zfs_vdev_cache_size 0 zfs_vdev_mirror_switch_us 10000 zfs_dirty_data_sync 67108864 spa_config_path /etc/zfs/zpool.cache zfs_dirty_data_max_max 4175752192 zfs_arc_lotsfree_percent 10 zfs_zevent_len_max 128 zfs_scan_min_time_ms 1000 zfs_arc_sys_free 0 zfs_arc_meta_strategy 1 zfs_vdev_cache_bshift 16 zfs_arc_meta_adjust_restarts 4096 zfs_max_recordsize 1048576 zfs_vdev_scrub_min_active 1 zfs_vdev_read_gap_limit 32768 zfs_arc_meta_limit 0 zfs_vdev_sync_write_max_active 10 l2arc_norw 0 zfs_arc_meta_prune 10000 metaslab_preload_enabled 1 l2arc_nocompress 0 zvol_major 230 zfs_vdev_aggregation_limit 131072 zfs_flags 0 spa_asize_inflation 24 zfs_admin_snapshot 0 l2arc_feed_secs 1 zio_taskq_batch_pct 75 zfs_sync_pass_deferred_free 2 zfs_disable_dup_eviction 0 zfs_arc_grow_retry 0 zfs_read_history_hits 0 zfs_vdev_async_write_min_active 1 zfs_vdev_async_read_max_active 3 zfs_scrub_delay 4 zfs_delay_min_dirty_percent 60 zfs_free_max_blocks 100000 zfs_vdev_cache_max 16384 zio_delay_max 30000 zfs_top_maxinflight 32 spa_slop_shift 5 zfs_vdev_write_gap_limit 4096 spa_load_verify_metadata 1 spa_load_verify_maxinflight 10000 l2arc_noprefetch 1 zfs_vdev_scheduler noop zfs_expire_snapshot 300 zfs_sync_pass_rewrite 2 zil_replay_disable 0 zfs_nocacheflush 0 zfs_arc_max 0 zfs_arc_min 0 zfs_read_chunk_size 1048576 zfs_txg_timeout 5 zfs_pd_bytes_max 52428800 l2arc_headroom_boost 200 zfs_send_corrupt_data 0 l2arc_feed_min_ms 200 zfs_arc_meta_min 0 zfs_arc_average_blocksize 8192 zfetch_array_rd_sz 1048576 zfs_autoimport_disable 1 zfs_arc_p_min_shift 0 zio_requeue_io_start_cut_in_line 1 zfs_vdev_sync_read_max_active 10 zfs_mdcomp_disable 0 zfs_arc_num_sublists_per_state 4 Both tools are well-documented and distributed upstream, see https://github.com/zfsonlinux/zfs/blob/master/cmd/arcstat/arcstat.py https://github.com/zfsonlinux/zfs/blob/master/cmd/arc_summary/arc_summary.py To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/zfs-linux/+bug/1574342/+subscriptions -- Mailing list: https://launchpad.net/~kernel-packages Post to : kernel-packages@lists.launchpad.net Unsubscribe : https://launchpad.net/~kernel-packages More help : https://help.launchpad.net/ListHelp