Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-17 Thread Mark Martinec
On 07/08/2018 15:58, Mark Martinec wrote: Collected, here it is:   https://www.ijs.si/usr/mark/tmp/dtrace-cmd.out.bz2 2018-08-14 11:18, Andriy Gapon wrote: I see one memory leak, not sure if it's the only one. It looks like vdev_geom_read_config() leaks all parsed vdev nvlist-s but the

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-14 Thread Andriy Gapon
On 07/08/2018 15:58, Mark Martinec wrote: > Collected, here it is: > >   https://www.ijs.si/usr/mark/tmp/dtrace-cmd.out.bz2 I see one memory leak, not sure if it's the only one. It looks like vdev_geom_read_config() leaks all parsed vdev nvlist-s but the last. The problems seems to come from

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-13 Thread Mark Martinec
2018-08-13 21:48, Volodymyr Kostyrko wrote: I've been in the same situation. ZFS, only pool, no ZFS errors. I think the problem is rather between swapping and ZFS ARC. This host has different load, sometimes it needs more active memory, somtimes less... This means that active zone can expand

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-13 Thread Volodymyr Kostyrko
23.07.18 18:12, Mark Martinec wrote: After upgrading an older AMD host from FreeBSD 10.3 to 11.1-RELEASE-p11 (amd64), ZFS is gradually eating up all memory, so that it crashes every few days when the memory is completely exhausted (after swapping heavily for a couple of hours). I've been in

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-07 Thread Mark Martinec
On Sat, Aug 04, 2018 at 08:38:04PM +0200, Mark Martinec wrote: 2018-08-04 19:01, Mark Johnston wrote: > I think running "zpool list" is adding a lot of noise to the output. > Could you retry without doing that? No, like I said previously, the "zpool list" (with one defunct zfs pool) *is* the

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-04 Thread Mark Johnston
On Sat, Aug 04, 2018 at 08:38:04PM +0200, Mark Martinec wrote: > 2018-08-04 19:01, Mark Johnston wrote: > > I think running "zpool list" is adding a lot of noise to the output. > > Could you retry without doing that? > > No, like I said previously, the "zpool list" (with one defunct > zfs pool)

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-04 Thread Mark Martinec
2018-08-04 19:01, Mark Johnston wrote: I think running "zpool list" is adding a lot of noise to the output. Could you retry without doing that? No, like I said previously, the "zpool list" (with one defunct zfs pool) *is* the sole culprit of the zfs memory leak. With each invocation of "zpool

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-04 Thread Mark Johnston
On Fri, Aug 03, 2018 at 09:11:42PM +0200, Mark Martinec wrote: > More attempts at tracking this down. The suggested dtrace command does > usually abort with: > >Assertion failed: (buf->dtbd_timestamp >= first_timestamp), > file >

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-03 Thread Mark Martinec
More attempts at tracking this down. The suggested dtrace command does usually abort with: Assertion failed: (buf->dtbd_timestamp >= first_timestamp), file /usr/src/cddl/contrib/opensolaris/lib/libdtrace/common/dt_consume.c, line 3330. but with some luck soon after each machine

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-08-01 Thread Mark Martinec
On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote: I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE and the situation has not improved. Also turned off all services. ZFS is still leaking memory about 30 MB per hour, until the host runs out of memory and swap space

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Nagy László Zsolt
I have experienced a very similar thing. After upgrading my machine from 11.1-R to 11.2-R, the swap space is filled up to about 66% in about every 2 days. First I tought that it was PostgreSQL, and lowered the shared_buffers setting, but it only postponed the problem for another day. The only

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Shane Ambler
On 01/08/2018 07:24, Mark Martinec wrote: > I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE > and the situation has not improved. Also turned off all services. > ZFS is still leaking memory about 30 MB per hour, until the host > runs out of memory and swap space and crashes,

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Mark Johnston
On Tue, Jul 31, 2018 at 11:54:29PM +0200, Mark Martinec wrote: > I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE > and the situation has not improved. Also turned off all services. > ZFS is still leaking memory about 30 MB per hour, until the host > runs out of memory and swap

Re: All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-31 Thread Mark Martinec
I have now upgraded this host from 11.1-RELEASE-p11 to 11.2-RELEASE and the situation has not improved. Also turned off all services. ZFS is still leaking memory about 30 MB per hour, until the host runs out of memory and swap space and crashes, unless I reboot it first every four days. Any

All the memory eaten away by ZFS 'solaris' malloc - on 11.1-R amd64

2018-07-23 Thread Mark Martinec
After upgrading an older AMD host from FreeBSD 10.3 to 11.1-RELEASE-p11 (amd64), ZFS is gradually eating up all memory, so that it crashes every few days when the memory is completely exhausted (after swapping heavily for a couple of hours). This machine has only 4 GB of memory. After capping up