I agree with Gaurav. We should try to identify what is causing current memory limit not to work.
If there is a need for a stop gap solution, we could temporarily make it higher. But I believe this is not a solution unless we answer the "why" question. ~ Chinmay. On Thu, Dec 24, 2015 at 10:23 AM, Gaurav Gupta <[email protected]> wrote: > Tim, > > This setting used to work fine earlier. What did we change/add in App > master that is causing this behavior. We should know the reason before > bumping the memory. May be there is some other issue. > > Thanks > - Gaurav > > > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <[email protected]> > wrote: > > > > I am ok with 1280MB. > > On 24 Dec 2015 06:34, "Timothy Farkas" <[email protected]> wrote: > > > >> If there are no objections I will open a pull request shortly to bump up > >> the default App Master Memory to 1280 MB. I recently got burned by this > >> default AGAIN when the App Master randomly died in the middle of demo. > The > >> demo application was small and only had 6 physical operators. > >> > >> Thanks, > >> Tim > >> > >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <[email protected] > > > >> wrote: > >> > >>> Sorry, meant to say 4-6GB, not MB. > >>> > >>> Ram > >>> > >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath < > [email protected]> > >>> wrote: > >>> > >>>> I'm OK with increasing it to 1280MB. > >>>> > >>>> I agree with Sandeep about new people trying out Apex, > >>>> often on limited environments like VMs limited to 4-6MB. With a > default > >>> of > >>>> 0.5GB per Buffer Server > >>>> and 1GB per operator, things start to fail when the user has more than > >> a > >>>> handful of operators. > >>>> > >>>> I'm guessing it's not just the number of the operators that puts > memory > >>>> pressure on master memory > >>>> but the size of the jars that get pulled in by all the operator > classes > >>>> and their dependencies. If this is > >>>> true, one could cause failure even with a single, sufficiently complex > >>>> operator. > >>>> > >>>> Ram > >>>> > >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <[email protected]> > >>>> wrote: > >>>> > >>>>> Hi Sandeep, > >>>>> > >>>>> I have seen the issue even with small apps like Twitter Demo. It > >> happens > >>>>> less frequently in very small apps but the issue is still there. It > >>>>> happens > >>>>> frequently in medium size apps (10 containers). Could we increase the > >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater > >>>>> stability with the defaults settings without allocating too much > >>>>> additional > >>>>> memory? > >>>>> > >>>>> Thanks, > >>>>> Tim > >>>>> > >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh < > >>>>> [email protected]> > >>>>> wrote: > >>>>> > >>>>>> With large number of containers, the problem becomes severe and I > >> too > >>>>> have > >>>>>> faced it. > >>>>>> > >>>>>> Not sure if default should be increased as that should work in most > >> of > >>>>> the > >>>>>> cases and even when new people are trying out Apex for the first > >> time. > >>>>>> They may not have enough memory. > >>>>>> > >>>>>> Regards > >>>>>> Sandeep > >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <[email protected]> > >>>>> wrote: > >>>>>> > >>>>>>> Tim - great point, we reduce the memory consumption for operators > >>> and > >>>>> do > >>>>>>> actually increase it for the app master, especially when we have > >>> more > >>>>>>> operators. I misread your original email. > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> Thank you, > >>>>>>> Ilya Ganelin > >>>>>>> > >>>>>>> > >>>>>>> > >>>>>>> -----Original Message----- > >>>>>>> From: Timothy Farkas [[email protected]<mailto: > >>> [email protected] > >>>>>> ] > >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time > >>>>>>> To: [email protected] > >>>>>>> Subject: Re: Default App Master Container Memory Size > >>>>>>> > >>>>>>> > >>>>>>> Hi Ilya, > >>>>>>> > >>>>>>> Dou you reduce the memory to 256 mb for operators only or for the > >>> App > >>>>>>> Master as well? > >>>>>>> > >>>>>>> Thanks, > >>>>>>> Tim > >>>>>>> > >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya < > >>>>>>> [email protected]> > >>>>>>> wrote: > >>>>>>> > >>>>>>>> Tim - in our applications we've actually tried to go with > >> smaller > >>>>>>>> containers, as small as 256 MB, simply because many operators > >>> don't > >>>>>>> demand > >>>>>>>> any memory and this allows us to run more partitions. > >>>>>>>> > >>>>>>>> We haven't noticed performance degradation. > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> Thank you, > >>>>>>>> Ilya Ganelin > >>>>>>>> > >>>>>>>> > >>>>>>>> > >>>>>>>> -----Original Message----- > >>>>>>>> From: Timothy Farkas [[email protected]<mailto: > >>>>> [email protected]>] > >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time > >>>>>>>> To: [email protected] > >>>>>>>> Subject: Default App Master Container Memory Size > >>>>>>>> > >>>>>>>> > >>>>>>>> Hi All, > >>>>>>>> > >>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed in > >>>>>> multiple > >>>>>>>> demo apps and in the production apps of some of our users that > >>> 1024 > >>>>> MB > >>>>>> is > >>>>>>>> not enough memory, and the app master dies. This was observed on > >>>>> small > >>>>>>>> applications with 5 - 20 containers. I think the default app > >>> master > >>>>>>> memory > >>>>>>>> size should be 2048. > >>>>>>>> > >>>>>>>> Any feedback? > >>>>>>>> > >>>>>>>> Thanks, > >>>>>>>> Tim > >>>>>>>> ________________________________________________________ > >>>>>>>> > >>>>>>>> The information contained in this e-mail is confidential and/or > >>>>>>>> proprietary to Capital One and/or its affiliates and may only be > >>>>> used > >>>>>>>> solely in performance of work or services for Capital One. The > >>>>>>> information > >>>>>>>> transmitted herewith is intended only for use by the individual > >> or > >>>>>> entity > >>>>>>>> to which it is addressed. If the reader of this message is not > >> the > >>>>>>> intended > >>>>>>>> recipient, you are hereby notified that any review, > >>> retransmission, > >>>>>>>> dissemination, distribution, copying or other use of, or taking > >> of > >>>>> any > >>>>>>>> action in reliance upon this information is strictly prohibited. > >>> If > >>>>> you > >>>>>>>> have received this communication in error, please contact the > >>> sender > >>>>>> and > >>>>>>>> delete the material from your computer. > >>>>>>>> > >>>>>>> ________________________________________________________ > >>>>>>> > >>>>>>> The information contained in this e-mail is confidential and/or > >>>>>>> proprietary to Capital One and/or its affiliates and may only be > >>> used > >>>>>>> solely in performance of work or services for Capital One. The > >>>>>> information > >>>>>>> transmitted herewith is intended only for use by the individual or > >>>>> entity > >>>>>>> to which it is addressed. If the reader of this message is not the > >>>>>> intended > >>>>>>> recipient, you are hereby notified that any review, > >> retransmission, > >>>>>>> dissemination, distribution, copying or other use of, or taking of > >>> any > >>>>>>> action in reliance upon this information is strictly prohibited. > >> If > >>>>> you > >>>>>>> have received this communication in error, please contact the > >> sender > >>>>> and > >>>>>>> delete the material from your computer. > >>>>>>> > >>>>>> > >>>>> > >>>> > >>>> > >>> > >> > >
