I've opened the following PR: https://github.com/openzfs/openzfs/pull/430
On Fri, Jul 21, 2017 at 12:30 PM, Prakash Surya <[email protected]> wrote: > Yuri, > > Looking at our internal bug tracker at Delphix, I see a bug reporting > these leaks: > > > ffffff0340abac18::bufctl -v > ADDR BUFADDR TIMESTAMP THREAD > CACHE LASTLOG CONTENTS > ffffff0340abac18 ffffff0340a6d8d0 b207a68c7d ffffff035fda93c0 > ffffff030e626588 ffffff0313c9c380 ffffff0317602cc8 > kmem_cache_alloc_debug+0x2e0 > kmem_cache_alloc+0xdd > kmem_alloc+0x4b > nv_alloc_sys+0x1f > nv_mem_zalloc+0x21 > nvlist_xalloc+0x67 > nvlist_alloc+0x2a > fnvlist_alloc+0x18 > get_clones_stat+0x30 > dsl_dataset_stats+0xcc > dmu_objset_stats+0x2e > zfs_ioc_objset_stats_impl+0x64 > zfs_ioc_snapshot_list_next+0x15c > zfsdev_ioctl+0x527 > cdev_ioctl+0x39 > > > ffffff03422d3290::bufctl -v > ADDR BUFADDR TIMESTAMP THREAD > CACHE LASTLOG CONTENTS > ffffff03422d3290 ffffff03422acf90 627f3f04a3d ffffff032aadd7c0 > ffffff030e628848 ffffff0310f74fc0 ffffff0319735b08 > kmem_cache_alloc_debug+0x2e0 > kmem_cache_alloc+0xdd > kmem_alloc+0x4b > nv_alloc_sys+0x1f > nv_mem_zalloc+0x21 > nvp_buf_alloc+0x24 > nvlist_add_common+0x11c > nvlist_copy_pairs+0x64 > nvlist_copy_embedded+0x50 > nvlist_add_common+0x36d > nvlist_add_nvlist+0x26 > fnvlist_add_nvlist+0x19 > spa_config_generate+0x1ff > spa_load_best+0xa1 > spa_import+0x1e2 > > Additionally, it looks like we fixed it back in October, but I don't see > this change on illumos: > > ± g show > commit 51a86fc8ebe33698ddbbf5f54f5ea0d566f05de6 (HEAD -> illumos-master) > Author: Pavel Zakharov <[email protected]> > Date: Thu Oct 6 11:13:12 2016 -0400 > > DLPX-47890 nvlist memory leak in get_clones_stat() and spa_load_best() > > diff --git a/usr/src/uts/common/fs/zfs/dsl_dataset.c > b/usr/src/uts/common/fs/zfs/dsl_dataset.c > index 81798db2fc..5706965ffe 100644 > --- a/usr/src/uts/common/fs/zfs/dsl_dataset.c > +++ b/usr/src/uts/common/fs/zfs/dsl_dataset.c > @@ -1742,10 +1742,10 @@ get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv) > fnvlist_add_nvlist(propval, ZPROP_VALUE, val); > fnvlist_add_nvlist(nv, zfs_prop_to_name(ZFS_PROP_CLONES), > propval); > - } else { > - nvlist_free(val); > - nvlist_free(propval); > } > + > + nvlist_free(val); > + nvlist_free(propval); > } > > /* > diff --git a/usr/src/uts/common/fs/zfs/spa.c b/usr/src/uts/common/fs/zfs/ > spa.c > index 8e72500722..4fa6a6c79a 100644 > --- a/usr/src/uts/common/fs/zfs/spa.c > +++ b/usr/src/uts/common/fs/zfs/spa.c > @@ -3024,6 +3024,8 @@ spa_load_best(spa_t *spa, spa_load_state_t state, > int mosconfig, > > if (config && (rewind_error || state != SPA_LOAD_RECOVER)) > spa_config_set(spa, config); > + else > + nvlist_free(config); > > if (state == SPA_LOAD_RECOVER) { > ASSERT3P(loadinfo, ==, NULL); > > I can open a PR to openzfs to upstream this patch, but can you also verify > this fixes your issue? The stacks looks the same to me, but I just want to > verify with you before I make an incorrect assumption. > > On Fri, Jul 21, 2017 at 11:38 AM, Yuri Pankov <[email protected]> wrote: > >> Hi, >> >> I'm not sure I've picked the change correctly, just guessing as it was >> the latest one to touch get_clones_stat() and friends, and if these aren't >> false positives.. but doing a `reboot -d` on current illumos-gate HEAD >> gives the following "::findleaks -dv" output: >> >> findleaks: maximum buffers => 301061 >> findleaks: actual buffers => 297587 >> findleaks: >> findleaks: potential pointers => 29289774 >> findleaks: dismissals => 26242305 (89.5%) >> findleaks: misses => 331153 ( 1.1%) >> findleaks: dups => 2419681 ( 8.2%) >> findleaks: follows => 296635 ( 1.0%) >> findleaks: >> findleaks: peak memory usage => 7353 kB >> findleaks: elapsed CPU time => 1.5 seconds >> findleaks: elapsed wall time => 2.0 seconds >> findleaks: >> CACHE LEAKED BUFCTL CALLER >> ffffff03d222b008 120 ffffff03ef7ceb78 nv_alloc_sys+0x1f >> ffffff03d222a448 123 ffffff03f4150cc8 nv_alloc_sys+0x1f >> ffffff03d222b448 5 ffffff03f28bd598 nv_alloc_sys+0x1f >> ffffff03d222b888 87 ffffff03f28c10f0 nv_alloc_sys+0x1f >> ffffff03d222c008 21 ffffff03f4139310 nv_alloc_sys+0x1f >> ffffff03d222b888 43 ffffff040ef3f3e8 nv_alloc_sys+0x1f >> ffffff03d222c008 120 ffffff03f4591e58 nv_alloc_sys+0x1f >> ffffff03d222b008 121 ffffff03f352c068 nv_alloc_sys+0x1f >> ffffff03d222a448 112 ffffff03f414e5f8 nv_alloc_sys+0x1f >> ffffff03d222b008 119 ffffff03ee92fdc0 nv_alloc_sys+0x1f >> ffffff03d222b888 46 ffffff03f28c1378 nv_alloc_sys+0x1f >> ffffff03d222b448 4 ffffff03f28c7708 nv_alloc_sys+0x1f >> ffffff03d222c008 20 ffffff03f2a6e7e8 nv_alloc_sys+0x1f >> ffffff03d222a448 11 ffffff03f414f018 nv_alloc_sys+0x1f >> ------------------------------------------------------------------------ >> Total 952 buffers, 40896 bytes >> >> kmem_alloc_40 leak: 120 buffers, 40 bytes each, 4800 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03ef7ceb78 ffffff03ed8e8c80 ab5dc8356 ffffff03f3b97100 >> ffffff03d222b008 ffffff03d9d4f5c0 ffffff03e57574d8 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nv_priv_alloc_embedded+0x1d >> nvlist_copy_embedded+0x2b >> nvlist_add_common+0x36d >> nvlist_add_nvlist+0x26 >> fnvlist_add_nvlist+0x19 >> get_clones_stat+0x83 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> kmem_alloc_24 leak: 123 buffers, 24 bytes each, 2952 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f4150cc8 ffffff03f43b29c0 ab5d77f2b ffffff03f3b97100 >> ffffff03d222a448 ffffff03d9d4a100 ffffff03e5756558 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvlist_xalloc+0x67 >> nvlist_alloc+0x2a >> fnvlist_alloc+0x18 >> get_clones_stat+0x27 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> zfsdev_ioctl+0x546 >> cdev_ioctl+0x39 >> >> kmem_alloc_48 leak: 5 buffers, 48 bytes each, 240 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f28bd598 ffffff03f28e5390 acfd4f020 ffffff03f3b97100 >> ffffff03d222b448 ffffff03db98a980 ffffff03de82d840 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_add_boolean+0x1f >> fnvlist_add_boolean+0x15 >> get_clones_stat_impl+0x151 >> get_clones_stat+0x4b >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> kmem_alloc_56 leak: 87 buffers, 56 bytes each, 4872 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f28c10f0 ffffff03f28e4a58 aba805885 ffffff03f3b97100 >> ffffff03d222b888 ffffff03d8ae5900 ffffff03e450d260 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_add_boolean+0x1f >> fnvlist_add_boolean+0x15 >> get_clones_stat_impl+0x151 >> get_clones_stat+0x4b >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> >> kmem_alloc_64 leak: 21 buffers, 64 bytes each, 1344 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f4139310 ffffff03f43c4500 ab5dbac69 ffffff03f3b97100 >> ffffff03d222c008 ffffff03d9d4ed80 ffffff03e5749298 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_add_boolean+0x1f >> fnvlist_add_boolean+0x15 >> get_clones_stat_impl+0x151 >> get_clones_stat+0x4b >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> kmem_alloc_56 leak: 43 buffers, 56 bytes each, 2408 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff040ef3f3e8 ffffff040fbc81f0 c8f9dddd9 ffffff040b1a23e0 >> ffffff03d222b888 ffffff03d7e13480 0 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0x320 >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_copy_pairs+0x64 >> nvlist_copy_embedded+0x50 >> nvlist_add_common+0x36d >> nvlist_add_nvlist+0x26 >> fnvlist_add_nvlist+0x19 >> get_clones_stat+0x83 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> >> kmem_alloc_64 leak: 120 buffers, 64 bytes each, 7680 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f4591e58 ffffff03f457be00 ab5dc7f46 ffffff03f3b97100 >> ffffff03d222c008 ffffff03d9d4f500 ffffff03e57491d0 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_add_nvlist+0x26 >> fnvlist_add_nvlist+0x19 >> get_clones_stat+0x83 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> zfsdev_ioctl+0x546 >> kmem_alloc_40 leak: 121 buffers, 40 bytes each, 4840 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f352c068 ffffff03f34bc840 ab5d77d47 ffffff03f3b97100 >> ffffff03d222b008 ffffff03d9d4a040 ffffff03e5756570 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_priv_alloc+0x1e >> nvlist_xalloc+0x4d >> nvlist_alloc+0x2a >> fnvlist_alloc+0x18 >> get_clones_stat+0x27 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> zfsdev_ioctl+0x546 >> cdev_ioctl+0x39 >> >> kmem_alloc_24 leak: 112 buffers, 24 bytes each, 2688 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f414e5f8 ffffff03f43b2480 abec5ac72 ffffff03f3b97100 >> ffffff03d222a448 ffffff03d5331700 ffffff03e302c208 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvlist_xalloc+0x67 >> nvlist_alloc+0x2a >> get_clones_stat+0x37 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> zfsdev_ioctl+0x546 >> cdev_ioctl+0x39 >> spec_ioctl+0x60 >> kmem_alloc_40 leak: 119 buffers, 40 bytes each, 4760 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03ee92fdc0 ffffff03eeee2dc0 ab5d782a6 ffffff03f3b97100 >> ffffff03d222b008 ffffff03d9d4a1c0 ffffff03e57490d0 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_priv_alloc+0x1e >> nvlist_xalloc+0x4d >> nvlist_alloc+0x2a >> get_clones_stat+0x37 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> zfsdev_ioctl+0x546 >> cdev_ioctl+0x39 >> spec_ioctl+0x60 >> >> kmem_alloc_56 leak: 46 buffers, 56 bytes each, 2576 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f28c1378 ffffff03f28e4968 aba80e3f6 ffffff03f3b97100 >> ffffff03d222b888 ffffff03d8ae6200 ffffff03e2945e98 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_copy_pairs+0x64 >> nvlist_copy_embedded+0x50 >> nvlist_add_common+0x36d >> nvlist_add_nvlist+0x26 >> fnvlist_add_nvlist+0x19 >> get_clones_stat+0x83 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> kmem_alloc_48 leak: 4 buffers, 48 bytes each, 192 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f28c7708 ffffff03f28e27d8 acfd5c6b6 ffffff03f3b97100 >> ffffff03d222b448 ffffff03db98b280 ffffff03de838288 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_copy_pairs+0x64 >> nvlist_copy_embedded+0x50 >> nvlist_add_common+0x36d >> nvlist_add_nvlist+0x26 >> fnvlist_add_nvlist+0x19 >> get_clones_stat+0x83 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> >> kmem_alloc_64 leak: 20 buffers, 64 bytes each, 1280 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f2a6e7e8 ffffff03f3ee4280 ab5dc893d ffffff03f3b97100 >> ffffff03d222c008 ffffff03d9d4f680 ffffff03e5748cd0 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0xdd >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvp_buf_alloc+0x24 >> nvlist_add_common+0x11c >> nvlist_copy_pairs+0x64 >> nvlist_copy_embedded+0x50 >> nvlist_add_common+0x36d >> nvlist_add_nvlist+0x26 >> fnvlist_add_nvlist+0x19 >> get_clones_stat+0x83 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> >> kmem_alloc_24 leak: 11 buffers, 24 bytes each, 264 bytes total >> ADDR BUFADDR TIMESTAMP THREAD >> CACHE LASTLOG CONTENTS >> ffffff03f414f018 ffffff03f43b2930 ab5d787f9 ffffff03f3b97100 >> ffffff03d222a448 ffffff03d9d4a280 0 >> kmem_cache_alloc_debug+0x2e0 >> kmem_cache_alloc+0x320 >> kmem_alloc+0x4b >> nv_alloc_sys+0x1f >> nv_mem_zalloc+0x21 >> nvlist_xalloc+0x67 >> nvlist_alloc+0x2a >> get_clones_stat+0x37 >> dsl_dataset_stats+0xcc >> dmu_objset_stats+0x2e >> zfs_ioc_objset_stats_impl+0x64 >> zfs_ioc_snapshot_list_next+0x191 >> zfsdev_ioctl+0x546 >> cdev_ioctl+0x39 >> spec_ioctl+0x60 > > ------------------------------------------ openzfs-developer Archives: https://openzfs.topicbox.com/groups/developer/discussions/T858afbc054d53c06-M9d05b1464c6808600e774883 Powered by Topicbox: https://topicbox.com
