Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-21 Thread Andriy Gapon
on 18/02/2014 15:47 Vitalij Satanivskij said the following: No checksume errors or any other errors found for now. Thank you again! Could you please send me an output of sysctl kstat | fgrep 'l2' from a system that has been patched and with a sufficiently long uptime? -- Andriy Gapon

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-21 Thread Vitalij Satanivskij
Dear Andriy, system uptime is 8 days, 20:26 Output: kstat.zfs.misc.arcstats.evict_l2_cached: 9771077767680 kstat.zfs.misc.arcstats.evict_l2_eligible: 3844577713152 kstat.zfs.misc.arcstats.evict_l2_ineligible: 8855320643072 kstat.zfs.misc.arcstats.l2_hits: 79824726

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-18 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, I'm testing you patch for sometime and looks like everything is ok. At last for 5 day of working any notisible memory leak wos not found. AG AG I've been able to spend some time on this issue. AG Could you please try the following patch? AG

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-18 Thread Andriy Gapon
on 18/02/2014 15:38 Vitalij Satanivskij said the following: Dear Andriy and FreeBSD community, I'm testing you patch for sometime and looks like everything is ok. At last for 5 day of working any notisible memory leak wos not found. Vitalij, thank you very much for testing! What about

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-18 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, No checksume errors or any other errors found for now. Andriy Gapon wrote: AG on 18/02/2014 15:38 Vitalij Satanivskij said the following: AG Dear Andriy and FreeBSD community, AG AG I'm testing you patch for sometime and looks like everything is ok. AG

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-14 Thread Andriy Gapon
on 11/02/2014 16:38 Vitalij Satanivskij said the following: Get first result's while testing l2 without compression Memory leak is not seen for now ( system working only 20 hours) but zfs stats saying that l2 degraded output of zfs-stats -L:

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-11 Thread Andriy Gapon
on 07/02/2014 11:11 Andriy Gapon said the following: on 05/02/2014 14:22 Vitalij Satanivskij said the following: Dear Andriy and FreeBSD community, Ok. I'm get coredump on panic. What else i need to do? Vitalij, Vladimir, I have been able to reproduce the leak at work, so now I have

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-11 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, For now I begin testing l2 cache without compression (with you path provided in last messages) in production. I will test the new patch on the test server first, and then if all is ok on one of the production servers. Andriy Gapon wrote: AG on 07/02/2014

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-11 Thread Vitalij Satanivskij
Get first result's while testing l2 without compression Memory leak is not seen for now ( system working only 20 hours) but zfs stats saying that l2 degraded output of zfs-stats -L: ZFS Subsystem Report

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-07 Thread Andriy Gapon
on 05/02/2014 14:22 Vitalij Satanivskij said the following: Dear Andriy and FreeBSD community, Ok. I'm get coredump on panic. What else i need to do? Vitalij, Vladimir, I have been able to reproduce the leak at work, so now I have full access to all debugging information that I need.

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-05 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, Andriy Gapon wrote: AG on 04/02/2014 19:10 Vitalij Satanivskij said the following: AG Dear Andriy and FreeBSD community, AG AG I'm aply patch and ofter few minutes of work get new panic AG AG screen shot on picture. AG AG

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-05 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, Ok. I'm get coredump on panic. What else i need to do? Vitalij Satanivskij wrote: VS Dear Andriy and FreeBSD community, VS VS Andriy Gapon wrote: VS AG on 04/02/2014 19:10 Vitalij Satanivskij said the following: VS AG Dear Andriy and FreeBSD community, VS

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-04 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, With patch system panic on boot. After remove cache device from pool system boot without problem. After this cache added again and sone kernel panic happened Screen shot of panic here http://i61.tinypic.com/30sbx2g.jpg Vitalij Satanivskij wrote: VS Dear

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-04 Thread Andriy Gapon
on 04/02/2014 12:08 Vitalij Satanivskij said the following: Dear Andriy and FreeBSD community, With patch system panic on boot. After remove cache device from pool system boot without problem. After this cache added again and sone kernel panic happened Screen shot of panic here

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-04 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, I'm aply patch and ofter few minutes of work get new panic screen shot on picture. http://i59.tinypic.com/sfctvc.jpg Andriy Gapon wrote: AG on 04/02/2014 12:08 Vitalij Satanivskij said the following: AG AG Dear Andriy and FreeBSD community, AG AG

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-02-04 Thread Andriy Gapon
on 04/02/2014 19:10 Vitalij Satanivskij said the following: Dear Andriy and FreeBSD community, I'm aply patch and ofter few minutes of work get new panic screen shot on picture. http://i59.tinypic.com/sfctvc.jpg Does this happen too early to get a crashdump? Do you have a chance to

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-31 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, Build world with path failed with error /usr/src/cddl/lib/libzpool/../../../sys/cddl/contrib/opensolaris/uts/common/fs/zfs/arc.c:4642:13: error: use of undeclared identifier 'l2hdr' ASSERT3P(l2hdr-b_tmp_cdata, ==, NULL);

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-30 Thread Andriy Gapon
on 28/01/2014 11:28 Vladimir Sharun said the following: Dear Andriy and FreeBSD community, After applying this path one of the systems runs fine (disk subsystem load low to moderate - 10-20% busy sustained), Then I saw this patch was merged to the HEAD and we apply it to the one of

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-15 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, Andriy Gapon wrote: AG on 14/01/2014 07:27 Vladimir Sharun said the following: AG Dear Andriy and FreeBSD community, AG AG I am not sure if the buffers are leaked somehow or if they are actually in use. AG It's one of the very few places where data buffers

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-15 Thread Andriy Gapon
on 15/01/2014 12:28 Vitalij Satanivskij said the following: Dear Andriy and FreeBSD community, Andriy Gapon wrote: AG on 14/01/2014 07:27 Vladimir Sharun said the following: AG Dear Andriy and FreeBSD community, AG AG I am not sure if the buffers are leaked somehow or if they are

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-15 Thread Vitalij Satanivskij
Dear Andriy and FreeBSD community, AG AG The first hunk of the patch is renaming of abl2 to l2hdr. AG So it.s ok just change trim_map_free(abl2-b_dev-l2ad_vdev, abl2-b_daddr, ab-b_size, 0); to

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-14 Thread Andriy Gapon
on 14/01/2014 07:27 Vladimir Sharun said the following: Dear Andriy and FreeBSD community, I am not sure if the buffers are leaked somehow or if they are actually in use. It's one of the very few places where data buffers are allocated without charging ARC. In all other places it's quite

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-06 Thread Andriy Gapon
on 06/01/2014 13:14 Vladimir Sharun said the following: Dear Andriy and FreeBSD community, I got the few minutes run for this dtrace hook; here's the output for 15 minutes run: http://pastebin.com/pKm9kLwa Does it explain something ? The following makes me suspect a problem with L2ARC

Re: ARC pressured out, how to control/stabilize ? (reformatted to text/plain)

2014-01-05 Thread Andriy Gapon
on 04/01/2014 14:50 Vladimir Sharun said the following: [snip] ARC: 28G Total, 2085M MFU, 20G MRU, 29M Anon, 1858M Header, 3855M Other [snip] ITEM SIZE LIMIT USED FREE REQ FAIL SLEEP [snip] zio_data_buf_131072: 131072, 0, 488217, 9,287155442, 0, 0