The DAG attribute to do so is MASTER_MEMORY_MB and by default it is 1GB. Can you please increase it?
On Fri, Mar 11, 2016 at 3:18 PM, Chandni Singh <chan...@datatorrent.com> wrote: > Hey Ilya, > > Can you please assign more memory to the App Master and check? > > Chandni > > On Fri, Mar 11, 2016 at 3:09 PM, Ganelin, Ilya < > ilya.gane...@capitalone.com> wrote: > >> Now with files: >> https://gist.github.com/ilganeli/7f770374113b40ffa18a >> >> From: "Ganelin, Ilya" <ilya.gane...@capitalone.com<mailto: >> ilya.gane...@capitalone.com>> >> Reply-To: "dev@apex.incubator.apache.org<mailto: >> dev@apex.incubator.apache.org>" <dev@apex.incubator.apache.org<mailto: >> dev@apex.incubator.apache.org>> >> Date: Friday, March 11, 2016 at 3:02 PM >> To: "dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org>" >> <dev@apex.incubator.apache.org<mailto:dev@apex.incubator.apache.org>> >> Subject: Stack overflow errors when launching job >> >> Hi guys – I’m running into a very frustrating issue where certain DAG >> configurations cause the following error log (attached). When this happens, >> my application even fails to launch. This does not seem to be a YARN issue >> since this occurs even with a relatively small number of partitions/memory. >> >> I’ve attached the input and output operators in question. >> I can get this to occur predictable by >> >> 1. Increasing the partition count on my input operator (reads from >> HDFS) - values above 20 cause this error >> 2. Increase the partition count on my output operator (writes to HDFS) >> - values above 20 cause this error >> 3. Set stream locality from the default to either thread local, node >> local, or container_local on the output operator >> >> This behavior is very frustrating as it’s preventing me from partitioning >> my HDFS I/O appropriately, thus allowing me to scale to higher throughputs. >> >> Do you have any thoughts on what’s going wrong? I would love your >> feedback. >> >> >> ________________________________ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> ________________________________________________________ >> >> The information contained in this e-mail is confidential and/or >> proprietary to Capital One and/or its affiliates and may only be used >> solely in performance of work or services for Capital One. The information >> transmitted herewith is intended only for use by the individual or entity >> to which it is addressed. If the reader of this message is not the intended >> recipient, you are hereby notified that any review, retransmission, >> dissemination, distribution, copying or other use of, or taking of any >> action in reliance upon this information is strictly prohibited. If you >> have received this communication in error, please contact the sender and >> delete the material from your computer. >> > >
