Yuri,
Looking at our internal bug tracker at Delphix, I see a bug reporting these
leaks:
> ffffff0340abac18::bufctl -v
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff0340abac18 ffffff0340a6d8d0 b207a68c7d ffffff035fda93c0
ffffff030e626588 ffffff0313c9c380 ffffff0317602cc8
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvlist_xalloc+0x67
nvlist_alloc+0x2a
fnvlist_alloc+0x18
get_clones_stat+0x30
dsl_dataset_stats+0xcc
dmu_objset_stats+0x2e
zfs_ioc_objset_stats_impl+0x64
zfs_ioc_snapshot_list_next+0x15c
zfsdev_ioctl+0x527
cdev_ioctl+0x39
> ffffff03422d3290::bufctl -v
ADDR BUFADDR TIMESTAMP THREAD
CACHE LASTLOG CONTENTS
ffffff03422d3290 ffffff03422acf90 627f3f04a3d ffffff032aadd7c0
ffffff030e628848 ffffff0310f74fc0 ffffff0319735b08
kmem_cache_alloc_debug+0x2e0
kmem_cache_alloc+0xdd
kmem_alloc+0x4b
nv_alloc_sys+0x1f
nv_mem_zalloc+0x21
nvp_buf_alloc+0x24
nvlist_add_common+0x11c
nvlist_copy_pairs+0x64
nvlist_copy_embedded+0x50
nvlist_add_common+0x36d
nvlist_add_nvlist+0x26
fnvlist_add_nvlist+0x19
spa_config_generate+0x1ff
spa_load_best+0xa1
spa_import+0x1e2
Additionally, it looks like we fixed it back in October, but I don't see
this change on illumos:
± g show
commit 51a86fc8ebe33698ddbbf5f54f5ea0d566f05de6 (HEAD -> illumos-master)
Author: Pavel Zakharov <[email protected]>
Date: Thu Oct 6 11:13:12 2016 -0400
DLPX-47890 nvlist memory leak in get_clones_stat() and spa_load_best()
diff --git a/usr/src/uts/common/fs/zfs/dsl_dataset.c
b/usr/src/uts/common/fs/zfs/dsl_dataset.c
index 81798db2fc..5706965ffe 100644
--- a/usr/src/uts/common/fs/zfs/dsl_dataset.c
+++ b/usr/src/uts/common/fs/zfs/dsl_dataset.c
@@ -1742,10 +1742,10 @@ get_clones_stat(dsl_dataset_t *ds, nvlist_t *nv)
fnvlist_add_nvlist(propval, ZPROP_VALUE, val);
fnvlist_add_nvlist(nv, zfs_prop_to_name(ZFS_PROP_CLONES),
propval);
- } else {
- nvlist_free(val);
- nvlist_free(propval);
}
+
+ nvlist_free(val);
+ nvlist_free(propval);
}
/*
diff --git a/usr/src/uts/common/fs/zfs/spa.c
b/usr/src/uts/common/fs/zfs/spa.c
index 8e72500722..4fa6a6c79a 100644
--- a/usr/src/uts/common/fs/zfs/spa.c
+++ b/usr/src/uts/common/fs/zfs/spa.c
@@ -3024,6 +3024,8 @@ spa_load_best(spa_t *spa, spa_load_state_t state, int
mosconfig,
if (config && (rewind_error || state != SPA_LOAD_RECOVER))
spa_config_set(spa, config);
+ else
+ nvlist_free(config);
if (state == SPA_LOAD_RECOVER) {
ASSERT3P(loadinfo, ==, NULL);
I can open a PR to openzfs to upstream this patch, but can you also verify
this fixes your issue? The stacks looks the same to me, but I just want to
verify with you before I make an incorrect assumption.
On Fri, Jul 21, 2017 at 11:38 AM, Yuri Pankov <[email protected]> wrote:
> Hi,
>
> I'm not sure I've picked the change correctly, just guessing as it was the
> latest one to touch get_clones_stat() and friends, and if these aren't
> false positives.. but doing a `reboot -d` on current illumos-gate HEAD
> gives the following "::findleaks -dv" output:
>
> findleaks: maximum buffers => 301061
> findleaks: actual buffers => 297587
> findleaks:
> findleaks: potential pointers => 29289774
> findleaks: dismissals => 26242305 (89.5%)
> findleaks: misses => 331153 ( 1.1%)
> findleaks: dups => 2419681 ( 8.2%)
> findleaks: follows => 296635 ( 1.0%)
> findleaks:
> findleaks: peak memory usage => 7353 kB
> findleaks: elapsed CPU time => 1.5 seconds
> findleaks: elapsed wall time => 2.0 seconds
> findleaks:
> CACHE LEAKED BUFCTL CALLER
> ffffff03d222b008 120 ffffff03ef7ceb78 nv_alloc_sys+0x1f
> ffffff03d222a448 123 ffffff03f4150cc8 nv_alloc_sys+0x1f
> ffffff03d222b448 5 ffffff03f28bd598 nv_alloc_sys+0x1f
> ffffff03d222b888 87 ffffff03f28c10f0 nv_alloc_sys+0x1f
> ffffff03d222c008 21 ffffff03f4139310 nv_alloc_sys+0x1f
> ffffff03d222b888 43 ffffff040ef3f3e8 nv_alloc_sys+0x1f
> ffffff03d222c008 120 ffffff03f4591e58 nv_alloc_sys+0x1f
> ffffff03d222b008 121 ffffff03f352c068 nv_alloc_sys+0x1f
> ffffff03d222a448 112 ffffff03f414e5f8 nv_alloc_sys+0x1f
> ffffff03d222b008 119 ffffff03ee92fdc0 nv_alloc_sys+0x1f
> ffffff03d222b888 46 ffffff03f28c1378 nv_alloc_sys+0x1f
> ffffff03d222b448 4 ffffff03f28c7708 nv_alloc_sys+0x1f
> ffffff03d222c008 20 ffffff03f2a6e7e8 nv_alloc_sys+0x1f
> ffffff03d222a448 11 ffffff03f414f018 nv_alloc_sys+0x1f
> ------------------------------------------------------------------------
> Total 952 buffers, 40896 bytes
>
> kmem_alloc_40 leak: 120 buffers, 40 bytes each, 4800 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03ef7ceb78 ffffff03ed8e8c80 ab5dc8356 ffffff03f3b97100
> ffffff03d222b008 ffffff03d9d4f5c0 ffffff03e57574d8
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nv_priv_alloc_embedded+0x1d
> nvlist_copy_embedded+0x2b
> nvlist_add_common+0x36d
> nvlist_add_nvlist+0x26
> fnvlist_add_nvlist+0x19
> get_clones_stat+0x83
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> kmem_alloc_24 leak: 123 buffers, 24 bytes each, 2952 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f4150cc8 ffffff03f43b29c0 ab5d77f2b ffffff03f3b97100
> ffffff03d222a448 ffffff03d9d4a100 ffffff03e5756558
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvlist_xalloc+0x67
> nvlist_alloc+0x2a
> fnvlist_alloc+0x18
> get_clones_stat+0x27
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> zfsdev_ioctl+0x546
> cdev_ioctl+0x39
>
> kmem_alloc_48 leak: 5 buffers, 48 bytes each, 240 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f28bd598 ffffff03f28e5390 acfd4f020 ffffff03f3b97100
> ffffff03d222b448 ffffff03db98a980 ffffff03de82d840
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_add_boolean+0x1f
> fnvlist_add_boolean+0x15
> get_clones_stat_impl+0x151
> get_clones_stat+0x4b
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> kmem_alloc_56 leak: 87 buffers, 56 bytes each, 4872 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f28c10f0 ffffff03f28e4a58 aba805885 ffffff03f3b97100
> ffffff03d222b888 ffffff03d8ae5900 ffffff03e450d260
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_add_boolean+0x1f
> fnvlist_add_boolean+0x15
> get_clones_stat_impl+0x151
> get_clones_stat+0x4b
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
>
> kmem_alloc_64 leak: 21 buffers, 64 bytes each, 1344 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f4139310 ffffff03f43c4500 ab5dbac69 ffffff03f3b97100
> ffffff03d222c008 ffffff03d9d4ed80 ffffff03e5749298
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_add_boolean+0x1f
> fnvlist_add_boolean+0x15
> get_clones_stat_impl+0x151
> get_clones_stat+0x4b
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> kmem_alloc_56 leak: 43 buffers, 56 bytes each, 2408 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff040ef3f3e8 ffffff040fbc81f0 c8f9dddd9 ffffff040b1a23e0
> ffffff03d222b888 ffffff03d7e13480 0
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0x320
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_copy_pairs+0x64
> nvlist_copy_embedded+0x50
> nvlist_add_common+0x36d
> nvlist_add_nvlist+0x26
> fnvlist_add_nvlist+0x19
> get_clones_stat+0x83
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
>
> kmem_alloc_64 leak: 120 buffers, 64 bytes each, 7680 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f4591e58 ffffff03f457be00 ab5dc7f46 ffffff03f3b97100
> ffffff03d222c008 ffffff03d9d4f500 ffffff03e57491d0
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_add_nvlist+0x26
> fnvlist_add_nvlist+0x19
> get_clones_stat+0x83
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> zfsdev_ioctl+0x546
> kmem_alloc_40 leak: 121 buffers, 40 bytes each, 4840 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f352c068 ffffff03f34bc840 ab5d77d47 ffffff03f3b97100
> ffffff03d222b008 ffffff03d9d4a040 ffffff03e5756570
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_priv_alloc+0x1e
> nvlist_xalloc+0x4d
> nvlist_alloc+0x2a
> fnvlist_alloc+0x18
> get_clones_stat+0x27
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> zfsdev_ioctl+0x546
> cdev_ioctl+0x39
>
> kmem_alloc_24 leak: 112 buffers, 24 bytes each, 2688 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f414e5f8 ffffff03f43b2480 abec5ac72 ffffff03f3b97100
> ffffff03d222a448 ffffff03d5331700 ffffff03e302c208
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvlist_xalloc+0x67
> nvlist_alloc+0x2a
> get_clones_stat+0x37
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> zfsdev_ioctl+0x546
> cdev_ioctl+0x39
> spec_ioctl+0x60
> kmem_alloc_40 leak: 119 buffers, 40 bytes each, 4760 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03ee92fdc0 ffffff03eeee2dc0 ab5d782a6 ffffff03f3b97100
> ffffff03d222b008 ffffff03d9d4a1c0 ffffff03e57490d0
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_priv_alloc+0x1e
> nvlist_xalloc+0x4d
> nvlist_alloc+0x2a
> get_clones_stat+0x37
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> zfsdev_ioctl+0x546
> cdev_ioctl+0x39
> spec_ioctl+0x60
>
> kmem_alloc_56 leak: 46 buffers, 56 bytes each, 2576 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f28c1378 ffffff03f28e4968 aba80e3f6 ffffff03f3b97100
> ffffff03d222b888 ffffff03d8ae6200 ffffff03e2945e98
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_copy_pairs+0x64
> nvlist_copy_embedded+0x50
> nvlist_add_common+0x36d
> nvlist_add_nvlist+0x26
> fnvlist_add_nvlist+0x19
> get_clones_stat+0x83
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> kmem_alloc_48 leak: 4 buffers, 48 bytes each, 192 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f28c7708 ffffff03f28e27d8 acfd5c6b6 ffffff03f3b97100
> ffffff03d222b448 ffffff03db98b280 ffffff03de838288
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_copy_pairs+0x64
> nvlist_copy_embedded+0x50
> nvlist_add_common+0x36d
> nvlist_add_nvlist+0x26
> fnvlist_add_nvlist+0x19
> get_clones_stat+0x83
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
>
> kmem_alloc_64 leak: 20 buffers, 64 bytes each, 1280 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f2a6e7e8 ffffff03f3ee4280 ab5dc893d ffffff03f3b97100
> ffffff03d222c008 ffffff03d9d4f680 ffffff03e5748cd0
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0xdd
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvp_buf_alloc+0x24
> nvlist_add_common+0x11c
> nvlist_copy_pairs+0x64
> nvlist_copy_embedded+0x50
> nvlist_add_common+0x36d
> nvlist_add_nvlist+0x26
> fnvlist_add_nvlist+0x19
> get_clones_stat+0x83
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
>
> kmem_alloc_24 leak: 11 buffers, 24 bytes each, 264 bytes total
> ADDR BUFADDR TIMESTAMP THREAD
> CACHE LASTLOG CONTENTS
> ffffff03f414f018 ffffff03f43b2930 ab5d787f9 ffffff03f3b97100
> ffffff03d222a448 ffffff03d9d4a280 0
> kmem_cache_alloc_debug+0x2e0
> kmem_cache_alloc+0x320
> kmem_alloc+0x4b
> nv_alloc_sys+0x1f
> nv_mem_zalloc+0x21
> nvlist_xalloc+0x67
> nvlist_alloc+0x2a
> get_clones_stat+0x37
> dsl_dataset_stats+0xcc
> dmu_objset_stats+0x2e
> zfs_ioc_objset_stats_impl+0x64
> zfs_ioc_snapshot_list_next+0x191
> zfsdev_ioctl+0x546
> cdev_ioctl+0x39
> spec_ioctl+0x60
------------------------------------------
openzfs-developer
Archives:
https://openzfs.topicbox.com/groups/developer/discussions/T858afbc054d53c06-Mf592a0265b0099eb6e0c2bb3
Powered by Topicbox: https://topicbox.com
