Tim, This setting used to work fine earlier. What did we change/add in App master that is causing this behavior. We should know the reason before bumping the memory. May be there is some other issue.
Thanks - Gaurav > On Dec 23, 2015, at 6:55 PM, Sandeep Deshmukh <[email protected]> wrote: > > I am ok with 1280MB. > On 24 Dec 2015 06:34, "Timothy Farkas" <[email protected]> wrote: > >> If there are no objections I will open a pull request shortly to bump up >> the default App Master Memory to 1280 MB. I recently got burned by this >> default AGAIN when the App Master randomly died in the middle of demo. The >> demo application was small and only had 6 physical operators. >> >> Thanks, >> Tim >> >> On Mon, Dec 14, 2015 at 2:01 PM, Munagala Ramanath <[email protected]> >> wrote: >> >>> Sorry, meant to say 4-6GB, not MB. >>> >>> Ram >>> >>> On Mon, Dec 14, 2015 at 2:00 PM, Munagala Ramanath <[email protected]> >>> wrote: >>> >>>> I'm OK with increasing it to 1280MB. >>>> >>>> I agree with Sandeep about new people trying out Apex, >>>> often on limited environments like VMs limited to 4-6MB. With a default >>> of >>>> 0.5GB per Buffer Server >>>> and 1GB per operator, things start to fail when the user has more than >> a >>>> handful of operators. >>>> >>>> I'm guessing it's not just the number of the operators that puts memory >>>> pressure on master memory >>>> but the size of the jars that get pulled in by all the operator classes >>>> and their dependencies. If this is >>>> true, one could cause failure even with a single, sufficiently complex >>>> operator. >>>> >>>> Ram >>>> >>>> On Mon, Dec 14, 2015 at 1:41 PM, Timothy Farkas <[email protected]> >>>> wrote: >>>> >>>>> Hi Sandeep, >>>>> >>>>> I have seen the issue even with small apps like Twitter Demo. It >> happens >>>>> less frequently in very small apps but the issue is still there. It >>>>> happens >>>>> frequently in medium size apps (10 containers). Could we increase the >>>>> MASTER_MEMORY_MB to 1280 instead of 2048? That way we can get greater >>>>> stability with the defaults settings without allocating too much >>>>> additional >>>>> memory? >>>>> >>>>> Thanks, >>>>> Tim >>>>> >>>>> On Sun, Dec 13, 2015 at 1:22 AM, Sandeep Deshmukh < >>>>> [email protected]> >>>>> wrote: >>>>> >>>>>> With large number of containers, the problem becomes severe and I >> too >>>>> have >>>>>> faced it. >>>>>> >>>>>> Not sure if default should be increased as that should work in most >> of >>>>> the >>>>>> cases and even when new people are trying out Apex for the first >> time. >>>>>> They may not have enough memory. >>>>>> >>>>>> Regards >>>>>> Sandeep >>>>>> On 13 Dec 2015 11:30, "Ganelin, Ilya" <[email protected]> >>>>> wrote: >>>>>> >>>>>>> Tim - great point, we reduce the memory consumption for operators >>> and >>>>> do >>>>>>> actually increase it for the app master, especially when we have >>> more >>>>>>> operators. I misread your original email. >>>>>>> >>>>>>> >>>>>>> >>>>>>> Thank you, >>>>>>> Ilya Ganelin >>>>>>> >>>>>>> >>>>>>> >>>>>>> -----Original Message----- >>>>>>> From: Timothy Farkas [[email protected]<mailto: >>> [email protected] >>>>>> ] >>>>>>> Sent: Saturday, December 12, 2015 11:30 PM Eastern Standard Time >>>>>>> To: [email protected] >>>>>>> Subject: Re: Default App Master Container Memory Size >>>>>>> >>>>>>> >>>>>>> Hi Ilya, >>>>>>> >>>>>>> Dou you reduce the memory to 256 mb for operators only or for the >>> App >>>>>>> Master as well? >>>>>>> >>>>>>> Thanks, >>>>>>> Tim >>>>>>> >>>>>>> On Sat, Dec 12, 2015 at 8:06 PM, Ganelin, Ilya < >>>>>>> [email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Tim - in our applications we've actually tried to go with >> smaller >>>>>>>> containers, as small as 256 MB, simply because many operators >>> don't >>>>>>> demand >>>>>>>> any memory and this allows us to run more partitions. >>>>>>>> >>>>>>>> We haven't noticed performance degradation. >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Thank you, >>>>>>>> Ilya Ganelin >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> -----Original Message----- >>>>>>>> From: Timothy Farkas [[email protected]<mailto: >>>>> [email protected]>] >>>>>>>> Sent: Saturday, December 12, 2015 11:02 PM Eastern Standard Time >>>>>>>> To: [email protected] >>>>>>>> Subject: Default App Master Container Memory Size >>>>>>>> >>>>>>>> >>>>>>>> Hi All, >>>>>>>> >>>>>>>> The default value for MASTER_MEMORY_MB is 1024. I've observed in >>>>>> multiple >>>>>>>> demo apps and in the production apps of some of our users that >>> 1024 >>>>> MB >>>>>> is >>>>>>>> not enough memory, and the app master dies. This was observed on >>>>> small >>>>>>>> applications with 5 - 20 containers. I think the default app >>> master >>>>>>> memory >>>>>>>> size should be 2048. >>>>>>>> >>>>>>>> Any feedback? >>>>>>>> >>>>>>>> Thanks, >>>>>>>> Tim >>>>>>>> ________________________________________________________ >>>>>>>> >>>>>>>> The information contained in this e-mail is confidential and/or >>>>>>>> proprietary to Capital One and/or its affiliates and may only be >>>>> used >>>>>>>> solely in performance of work or services for Capital One. The >>>>>>> information >>>>>>>> transmitted herewith is intended only for use by the individual >> or >>>>>> entity >>>>>>>> to which it is addressed. If the reader of this message is not >> the >>>>>>> intended >>>>>>>> recipient, you are hereby notified that any review, >>> retransmission, >>>>>>>> dissemination, distribution, copying or other use of, or taking >> of >>>>> any >>>>>>>> action in reliance upon this information is strictly prohibited. >>> If >>>>> you >>>>>>>> have received this communication in error, please contact the >>> sender >>>>>> and >>>>>>>> delete the material from your computer. >>>>>>>> >>>>>>> ________________________________________________________ >>>>>>> >>>>>>> The information contained in this e-mail is confidential and/or >>>>>>> proprietary to Capital One and/or its affiliates and may only be >>> used >>>>>>> solely in performance of work or services for Capital One. The >>>>>> information >>>>>>> transmitted herewith is intended only for use by the individual or >>>>> entity >>>>>>> to which it is addressed. If the reader of this message is not the >>>>>> intended >>>>>>> recipient, you are hereby notified that any review, >> retransmission, >>>>>>> dissemination, distribution, copying or other use of, or taking of >>> any >>>>>>> action in reliance upon this information is strictly prohibited. >> If >>>>> you >>>>>>> have received this communication in error, please contact the >> sender >>>>> and >>>>>>> delete the material from your computer. >>>>>>> >>>>>> >>>>> >>>> >>>> >>> >>
