Re: [zones-discuss] ZFS ARC cache issue

[Nicolas Dorfsman]

When I looked for references on ARC freeing algo, I did find some lines 
of codes talking about freeing ARC when memory is under pressure.

Nice...but what could be memory under pressure in the kernel syntax ?


Jumping from C lines to blogs to docsI went back to basics :

- lotsfree
- fastscan


IMHO the lotsfree (The greater of 1/64th of physical memory or 512 
Kbytes) is stupid when you're using ZFS.



Nicolas
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[James Carlson]

On 06/06/10 18:58, Brad Diggs wrote:
> Apps that attempt to acquire a large amount of data on startup can fail
> to start properly if ZFS does not free
> up the memory fast enough.  I have observed this with Directory Server.

"Fast enough?"  The model I was expecting here was a bit more
synchronous than that -- someone asks for memory, the memory subsystem
asks ZFS to release some, and it does, which allows the request to be
satisfied.

I'd be interesting to know how time gets involved here at all.  If it
does, then I think that would make the "ZFS ARC should use all of memory
by default" scheme fairly problematic, as it would just intermittently
shoot down large processes.

Or just more fundamentally: needing to tune this at all seems a bit like
a bug to me.

-- 
James Carlson 42.703N 71.076W 
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Ketan]

Thanks to all you guys to give me more insight on the ZFS cache .. we had 
rebooted the system over the weekend and the zfs arc cache now sits at 5.7G 
only and fusion guys were able to start their application after the reboot.

Thanks again to all.
-- 
This message posted from opensolaris.org
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Brad Diggs]

Apps that attempt to acquire a large amount of data on startup can fail to start properly if ZFS does not freeup the memory fast enough.  I have observed this with Directory Server.  The best approach that I have found is to understand how you want RAM used on the system and then configure the ZFS ARC and respective applications to use set amounts such that there is no overlap between them.  See the relevantsections from my blog post on filesystem caching strategies for more information.Brad Brad Diggs | Principal Security Sales Consultant | +1.972.814.3698Oracle North America Technology Organization16000 Dallas Parkway, Dallas, TX 75248eMail: brad.di...@oracle.comTech Blog: http://TheZoneManager.comLinkedIn: http://www.linkedin.com/in/braddiggs On Jun 4, 2010, at 1:47 PM, James Carlson wrote:Ketan wrote:Let me know what command you want me to run on it for kstat  /truss  as per kstat zfs:0:zrcstats:size the size is approximately 40GSince there are a bunch of ways that the problem that Jason King wasdescribing could manifest, I think the only way to do this would be toget the system in a state where Fusion consistently fails to run, andthen start it up with:	truss -fo /tmp/fusion.out fusion-command-and-args...You'd then have to grovel through the /tmp/fusion.out and find out whatleads up to the failure and see if there's anything suspicious there.Since Fusion is Oracle and OpenSolaris and ZFS are Oracle, maybe there'sanother possibility.  This could be one of those cases where thathoped-for "synergy" might kick in.  ;-}-- James Carlson 42.703N 71.076W ___zones-discuss mailing listzones-discuss@opensolaris.org___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[James Carlson]

Ketan wrote:
> Let me know what command you want me to run on it for kstat  /truss  
>  
> as per kstat zfs:0:zrcstats:size the size is approximately 40G

Since there are a bunch of ways that the problem that Jason King was
describing could manifest, I think the only way to do this would be to
get the system in a state where Fusion consistently fails to run, and
then start it up with:

truss -fo /tmp/fusion.out fusion-command-and-args...

You'd then have to grovel through the /tmp/fusion.out and find out what
leads up to the failure and see if there's anything suspicious there.

Since Fusion is Oracle and OpenSolaris and ZFS are Oracle, maybe there's
another possibility.  This could be one of those cases where that
hoped-for "synergy" might kick in.  ;-}

-- 
James Carlson 42.703N 71.076W 
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Ketan]

Let me know what command you want me to run on it for kstat  /truss  
 
as per kstat zfs:0:zrcstats:size the size is approximately 40G
-- 
This message posted from opensolaris.org
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Jason King]

Are you sure fusion isn't checking the amount of available memory
itself and just deciding to abort?

It wouldn't be unprecendeted -- if you run Oracle RDBMS on NFS mounts,
it refuses to start unless it sees explicit mount options provided for
the database filesystems (even when they are merely affirming the
default behavior).

If you can, I'd try using truss or such -- I'd be interested to see if
it's running vmstat or looking at some kstats.


On Fri, Jun 4, 2010 at 11:04 AM, James Carlson  wrote:
> Petr Benes wrote:
>>> That leaves unanswered the underlying question: why do you need to do
>>> this at all?  Isn't the ZFS ARC supposed to release memory when the
>>> system is under pressure?  Is that mechanism not working well in some
>>> cases ... ?
>>
>> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6522017
>>
>> " ... Even if the ZFS ARC subsequently frees memory, the kernel cage does
>> not shrink.  It cannot shrink, because pages from the ZFS ARC were
>> interspersed with other kernel pages, so the free space in the
>> physical address range of the cage is fragmented when the ZFS pages
>> are released.  The remaining kernel pages cannot be moved to compress
>> the cage, as kernel memory inside the cage is not relocatable. ..."
>
> Sure ... but that refers specifically to DR-related issues, and that's
> not what the original poster complained about.  His original message
> said that he was having trouble with a large application (Oracle Fusion)
> running on a system using ZFS.  Does Fusion really need contiguous
> kernel memory (why?) or is there something else going on here?
>
> --
> James Carlson         42.703N 71.076W         
> ___
> zones-discuss mailing list
> zones-discuss@opensolaris.org
>
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Petr Benes]

> Sure ... but that refers specifically to DR-related issues,

DR-related issues with kernel cage unable to return memory.
In case you are on a DR-capable system you have troubles with
DR itself. On other HW kernel won't just return memory to OS.


> and that's
> not what the original poster complained about.  His original message
> said that he was having trouble with a large application (Oracle Fusion)
> running on a system using ZFS.  Does Fusion really need contiguous
> kernel memory (why?) or is there something else going on here?
>
> --
> James Carlson 42.703N 71.076W 
>
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[James Carlson]

Petr Benes wrote:
>> That leaves unanswered the underlying question: why do you need to do
>> this at all?  Isn't the ZFS ARC supposed to release memory when the
>> system is under pressure?  Is that mechanism not working well in some
>> cases ... ?
> 
> http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6522017
> 
> " ... Even if the ZFS ARC subsequently frees memory, the kernel cage does
> not shrink.  It cannot shrink, because pages from the ZFS ARC were
> interspersed with other kernel pages, so the free space in the
> physical address range of the cage is fragmented when the ZFS pages
> are released.  The remaining kernel pages cannot be moved to compress
> the cage, as kernel memory inside the cage is not relocatable. ..."

Sure ... but that refers specifically to DR-related issues, and that's
not what the original poster complained about.  His original message
said that he was having trouble with a large application (Oracle Fusion)
running on a system using ZFS.  Does Fusion really need contiguous
kernel memory (why?) or is there something else going on here?

-- 
James Carlson 42.703N 71.076W 
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Petr Benes]

> That leaves unanswered the underlying question: why do you need to do
> this at all?  Isn't the ZFS ARC supposed to release memory when the
> system is under pressure?  Is that mechanism not working well in some
> cases ... ?

http://bugs.opensolaris.org/bugdatabase/view_bug.do?bug_id=6522017

" ... Even if the ZFS ARC subsequently frees memory, the kernel cage does
not shrink.  It cannot shrink, because pages from the ZFS ARC were
interspersed with other kernel pages, so the free space in the
physical address range of the cage is fragmented when the ZFS pages
are released.  The remaining kernel pages cannot be moved to compress
the cage, as kernel memory inside the cage is not relocatable. ..."

Petr
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[James Carlson]

Petr Benes wrote:
> add to /etc/system something like (value depends on your needs)
> 
> * limit greedy ZFS to 4 GiB
> set zfs:zfs_arc_max = 4294967296
> 
> And yes, this has nothing to do with zones :-).

That leaves unanswered the underlying question: why do you need to do
this at all?  Isn't the ZFS ARC supposed to release memory when the
system is under pressure?  Is that mechanism not working well in some
cases ... ?

-- 
James Carlson 42.703N 71.076W 
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Petr Benes]

add to /etc/system something like (value depends on your needs)

* limit greedy ZFS to 4 GiB
set zfs:zfs_arc_max = 4294967296

And yes, this has nothing to do with zones :-).

Regards,
Petr

On 03/06/2010, Ketan  wrote:
> We are having a server running zfs root with 64G RAM and the system has 3
> zones running oracle fusion app and zfs cache is using 40G memory as per
>
> kstat zfs:0:arcstats:size.   and system shows only 5G of memory is free rest
> is taken by kernel and 2 remaining zones.
>
> Now my problem is that fusion guys are getting not enough memory message
> while starting their application due to top and vmstat shows 5G as free
> memory. But i read ZFS cache releases memory as required by the application
> so why fusion application is not starting up. Is there some we can do to
> decrease the ARC Cache usage on the fly without rebooting the global zone ?
> --
> This message posted from opensolaris.org
> ___
> zones-discuss mailing list
> zones-discuss@opensolaris.org
>
___
zones-discuss mailing list
zones-discuss@opensolaris.org

Re: [zones-discuss] ZFS ARC cache issue

[Steve Lawrence]

Try zfs-discuss.


Ketan wrote:
We are having a server running zfs root with 64G RAM and the system has 3 zones running oracle fusion app and zfs cache is using 40G memory as per 

kstat zfs:0:arcstats:size.   and system shows only 5G of memory is free rest is taken by kernel and 2 remaining zones. 


Now my problem is that fusion guys are getting not enough memory message while 
starting their application due to top and vmstat shows 5G as free memory. But i 
read ZFS cache releases memory as required by the application so why fusion 
application is not starting up. Is there some we can do to decrease the ARC 
Cache usage on the fly without rebooting the global zone ?
  

___
zones-discuss mailing list
zones-discuss@opensolaris.org

[zones-discuss] ZFS ARC cache issue

[Ketan]

We are having a server running zfs root with 64G RAM and the system has 3 zones 
running oracle fusion app and zfs cache is using 40G memory as per 

kstat zfs:0:arcstats:size.   and system shows only 5G of memory is free rest is 
taken by kernel and 2 remaining zones. 

Now my problem is that fusion guys are getting not enough memory message while 
starting their application due to top and vmstat shows 5G as free memory. But i 
read ZFS cache releases memory as required by the application so why fusion 
application is not starting up. Is there some we can do to decrease the ARC 
Cache usage on the fly without rebooting the global zone ?
-- 
This message posted from opensolaris.org
___
zones-discuss mailing list
zones-discuss@opensolaris.org