on 18/02/2014 15:47 Vitalij Satanivskij said the following:
No checksume errors or any other errors found for now.
Thank you again!
Could you please send me an output of
sysctl kstat | fgrep 'l2'
from a system that has been patched and with a sufficiently long uptime?
--
Andriy Gapon
Dear Andriy,
system uptime is 8 days, 20:26
Output:
kstat.zfs.misc.arcstats.evict_l2_cached: 9771077767680
kstat.zfs.misc.arcstats.evict_l2_eligible: 3844577713152
kstat.zfs.misc.arcstats.evict_l2_ineligible: 8855320643072
kstat.zfs.misc.arcstats.l2_hits: 79824726
Dear Andriy and FreeBSD community,
I'm testing you patch for sometime and looks like everything is ok.
At last for 5 day of working any notisible memory leak wos not found.
AG
AG I've been able to spend some time on this issue.
AG Could you please try the following patch?
AG
on 18/02/2014 15:38 Vitalij Satanivskij said the following:
Dear Andriy and FreeBSD community,
I'm testing you patch for sometime and looks like everything is ok.
At last for 5 day of working any notisible memory leak wos not found.
Vitalij,
thank you very much for testing!
What about
Dear Andriy and FreeBSD community,
No checksume errors or any other errors found for now.
Andriy Gapon wrote:
AG on 18/02/2014 15:38 Vitalij Satanivskij said the following:
AG Dear Andriy and FreeBSD community,
AG
AG I'm testing you patch for sometime and looks like everything is ok.
AG
on 11/02/2014 16:38 Vitalij Satanivskij said the following:
Get first result's while testing l2 without compression
Memory leak is not seen for now ( system working only 20 hours) but
zfs stats saying that l2 degraded
output of zfs-stats -L:
on 07/02/2014 11:11 Andriy Gapon said the following:
on 05/02/2014 14:22 Vitalij Satanivskij said the following:
Dear Andriy and FreeBSD community,
Ok. I'm get coredump on panic.
What else i need to do?
Vitalij, Vladimir,
I have been able to reproduce the leak at work, so now I have
Dear Andriy and FreeBSD community,
For now I begin testing l2 cache without compression (with you path provided in
last messages) in production.
I will test the new patch on the test server first, and then if all is ok on
one of the production servers.
Andriy Gapon wrote:
AG on 07/02/2014
Get first result's while testing l2 without compression
Memory leak is not seen for now ( system working only 20 hours) but
zfs stats saying that l2 degraded
output of zfs-stats -L:
ZFS Subsystem Report
on 05/02/2014 14:22 Vitalij Satanivskij said the following:
Dear Andriy and FreeBSD community,
Ok. I'm get coredump on panic.
What else i need to do?
Vitalij, Vladimir,
I have been able to reproduce the leak at work, so now I have full access to all
debugging information that I need.
Dear Andriy and FreeBSD community,
Andriy Gapon wrote:
AG on 04/02/2014 19:10 Vitalij Satanivskij said the following:
AG Dear Andriy and FreeBSD community,
AG
AG I'm aply patch and ofter few minutes of work get new panic
AG
AG screen shot on picture.
AG
AG
Dear Andriy and FreeBSD community,
Ok. I'm get coredump on panic.
What else i need to do?
Vitalij Satanivskij wrote:
VS Dear Andriy and FreeBSD community,
VS
VS Andriy Gapon wrote:
VS AG on 04/02/2014 19:10 Vitalij Satanivskij said the following:
VS AG Dear Andriy and FreeBSD community,
VS
Dear Andriy and FreeBSD community,
With patch system panic on boot.
After remove cache device from pool system boot without problem.
After this cache added again and sone kernel panic happened
Screen shot of panic here http://i61.tinypic.com/30sbx2g.jpg
Vitalij Satanivskij wrote:
VS Dear
on 04/02/2014 12:08 Vitalij Satanivskij said the following:
Dear Andriy and FreeBSD community,
With patch system panic on boot.
After remove cache device from pool system boot without problem.
After this cache added again and sone kernel panic happened
Screen shot of panic here
Dear Andriy and FreeBSD community,
I'm aply patch and ofter few minutes of work get new panic
screen shot on picture.
http://i59.tinypic.com/sfctvc.jpg
Andriy Gapon wrote:
AG on 04/02/2014 12:08 Vitalij Satanivskij said the following:
AG
AG Dear Andriy and FreeBSD community,
AG
AG
on 04/02/2014 19:10 Vitalij Satanivskij said the following:
Dear Andriy and FreeBSD community,
I'm aply patch and ofter few minutes of work get new panic
screen shot on picture.
http://i59.tinypic.com/sfctvc.jpg
Does this happen too early to get a crashdump?
Do you have a chance to
Dear Andriy and FreeBSD community,
Build world with path failed with error
/usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4642:13:
error: use of
undeclared identifier 'l2hdr'
ASSERT3P(l2hdr-b_tmp_cdata, ==, NULL);
on 28/01/2014 11:28 Vladimir Sharun said the following:
Dear Andriy and FreeBSD community,
After applying this path one of the systems runs fine (disk subsystem load
low to moderate
- 10-20% busy sustained),
Then I saw this patch was merged to the HEAD and we apply it to the one of
Dear Andriy and FreeBSD community,
L2ARC temporarily turned off by setting secondarycache=none everywhere it was
enabled,
so no more leak for one particular day.
Here's the top header:
last pid: 89916; load averages: 2.49, 2.91, 2.89up 5+19:21:42 14:09:12
561 processes: 2 running, 559
Dear Andriy and FreeBSD community,
After applying this path one of the systems runs fine (disk subsystem load low
to moderate
- 10-20% busy sustained),
Then I saw this patch was merged to the HEAD and we apply it to the one of the
systems
with moderate to high disk load: 30-60% busy
Dear Andriy and FreeBSD community,
Andriy Gapon wrote:
AG on 14/01/2014 07:27 Vladimir Sharun said the following:
AG Dear Andriy and FreeBSD community,
AG
AG I am not sure if the buffers are leaked somehow or if they are actually
in use.
AG It's one of the very few places where data buffers
on 15/01/2014 12:28 Vitalij Satanivskij said the following:
Dear Andriy and FreeBSD community,
Andriy Gapon wrote:
AG on 14/01/2014 07:27 Vladimir Sharun said the following:
AG Dear Andriy and FreeBSD community,
AG
AG I am not sure if the buffers are leaked somehow or if they are
Dear Andriy and FreeBSD community,
AG
AG The first hunk of the patch is renaming of abl2 to l2hdr.
AG
So it.s ok just change
trim_map_free(abl2-b_dev-l2ad_vdev, abl2-b_daddr,
ab-b_size, 0);
to
on 14/01/2014 07:27 Vladimir Sharun said the following:
Dear Andriy and FreeBSD community,
I am not sure if the buffers are leaked somehow or if they are actually in
use.
It's one of the very few places where data buffers are allocated without
charging ARC. In all other places it's quite
Dear Andriy and FreeBSD community,
I am not sure if the buffers are leaked somehow or if they are actually in
use.
It's one of the very few places where data buffers are allocated without
charging ARC. In all other places it's quite easy to match allocations and
deallocations. But in
Dear Andriy and FreeBSD community,
I got the few minutes run for this dtrace hook; here's the output for 15
minutes run:
http://pastebin.com/pKm9kLwa
Does it explain something ?
on 04/01/2014 14:50 Vladimir Sharun said the following:
[snip]
ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon,
on 06/01/2014 13:14 Vladimir Sharun said the following:
Dear Andriy and FreeBSD community,
I got the few minutes run for this dtrace hook; here's the output for 15
minutes run:
http://pastebin.com/pKm9kLwa
Does it explain something ?
The following makes me suspect a problem with L2ARC
Dear Andriy, FreeBSD community,
Thank you for your suggestion, so we will turn off the L2ARC and will report if
the issue persist or not.
For now the test server rebooted with L2ARC turned off and there's no
allocations done in
l2arc_feed_thread according to dtrace hook you provided.
Let's
on 04/01/2014 14:50 Vladimir Sharun said the following:
[snip]
ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other
[snip]
ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP
[snip]
zio_data_buf_131072: 131072, 0, 488217, 9,287155442, 0, 0
Good day community,
We ran the system in production with r259544 on it. 128Gb RAM, 72Gb - arc_max
and 42Gb - arc_min set in loader.conf.
The system ran both application (which are FS ops hungry) and 18Tb storage on
this setup.
From the start, ARC clearly shows 72Gb of memory consumed, but
30 matches
Mail list logo
