I'm OK with increasing it to 1280MB. I agree with Sandeep about new people trying out Apex, often on limited environments like VMs limited to 4-6MB. With a default of 0.5GB per Buffer Server and 1GB per operator, things start to fail when the user has more than a handful of operators.
I'm guessing it's not just the number of the operators that puts memory pressure on master memory but the size of the jars that get pulled in by all the operator classes and their dependencies. If this is true, one could cause failure even with a single, sufficiently complex operator. Ram On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <[email protected]> wrote: > Hi Sandeep, > > I have seen the issue even with small apps like Twitter Demo. It happens > less frequently in very small apps but the issue is still there. It happens > frequently in medium size apps (10 containers). Could we increase the > MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater > stability with the defaults settings without allocating too much additional > memory? > > Thanks, > Tim > > On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh <[email protected] > > > wrote: > > > With large number of containers, the problem becomes severe and I too > have > > faced it. > > > > Not sure if default should be increased as that should work in most of > the > > cases and even when new people are trying out Apex for the first time. > > They may not have enough memory. > > > > Regards > > Sandeep > > On 13 Dec 2015 11:30, "Ganelin, Ilya" <[email protected]> > wrote: > > > > > Tim - great point, we reduce the memory consumption for operators and > do > > > actually increase it for the app master, especially when we have more > > > operators. I misread your original email. > > > > > > > > > > > > Thank you, > > > Ilya Ganelin > > > > > > > > > > > > -----Original Message----- > > > From: Timothy Farkas [[email protected]<mailto:[email protected]>] > > > Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time > > > To: [email protected] > > > Subject: Re: Default App Master Container Memory Size > > > > > > > > > Hi Ilya, > > > > > > Dou you reduce the memory to 256 mb for operators only or for the App > > > Master as well? > > > > > > Thanks, > > > Tim > > > > > > On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya < > > > [email protected]> > > > wrote: > > > > > > > Tim - in our applications we've actually tried to go with smaller > > > > containers, as small as 256 MB, simply because many operators don't > > > demand > > > > any memory and this allows us to run more partitions. > > > > > > > > We haven't noticed performance degradation. > > > > > > > > > > > > > > > > Thank you, > > > > Ilya Ganelin > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Timothy Farkas [[email protected]<mailto:[email protected] > >] > > > > Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time > > > > To: [email protected] > > > > Subject: Default App Master Container Memory Size > > > > > > > > > > > > Hi All, > > > > > > > > The default value for MASTER_MEMORY_MB is 1024. I've observed in > > multiple > > > > demo apps and in the production apps of some of our users that 1024 > MB > > is > > > > not enough memory, and the app master dies. This was observed on > small > > > > applications with 5 - 20 containers. I think the default app master > > > memory > > > > size should be 2048. > > > > > > > > Any feedback? > > > > > > > > Thanks, > > > > Tim > > > > ________________________________________________________ > > > > > > > > The information contained in this e-mail is confidential and/or > > > > proprietary to Capital One and/or its affiliates and may only be used > > > > solely in performance of work or services for Capital One. The > > > information > > > > transmitted herewith is intended only for use by the individual or > > entity > > > > to which it is addressed. If the reader of this message is not the > > > intended > > > > recipient, you are hereby notified that any review, retransmission, > > > > dissemination, distribution, copying or other use of, or taking of > any > > > > action in reliance upon this information is strictly prohibited. If > you > > > > have received this communication in error, please contact the sender > > and > > > > delete the material from your computer. > > > > > > > ________________________________________________________ > > > > > > The information contained in this e-mail is confidential and/or > > > proprietary to Capital One and/or its affiliates and may only be used > > > solely in performance of work or services for Capital One. The > > information > > > transmitted herewith is intended only for use by the individual or > entity > > > to which it is addressed. If the reader of this message is not the > > intended > > > recipient, you are hereby notified that any review, retransmission, > > > dissemination, distribution, copying or other use of, or taking of any > > > action in reliance upon this information is strictly prohibited. If you > > > have received this communication in error, please contact the sender > and > > > delete the material from your computer. > > > > > >
